[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:13 dist-test-slave-cx8d krb5kdc[17686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:13 dist-test-slave-cx8d krb5kdc[17686](info): set up 2 sockets
Jun 20 21:55:13 dist-test-slave-cx8d krb5kdc[17686](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 21:55:14.302093 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:14 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:14.423244 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41005
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:14.532795 17699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:14.533084 17699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:14.533126 17699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:14.537190 17699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:14.537262 17699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:14.537283 17699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:14.537300 17699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:14.537315 17699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:14.543457 17699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:14.544600 17699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:14.545706 17699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:14.554661 17704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:14.554677 17707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:14.554747 17705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:14.555068 17699 server_base.cc:1034] running on GCE node
I20260620 21:55:14.555688 17699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:14.556958 17699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:14.558344 17699 hybrid_clock.cc:648] HybridClock initialized: now 1781992514558309 us; error 47 us; skew 500 ppm
Jun 20 21:55:14 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992514, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:14.562167 17699 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:14.563215 17699 webserver.cc:460] Webserver started at http://127.17.67.254:44213/ using document root <none> and password file <none>
I20260620 21:55:14.563834 17699 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:14.563930 17699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:14.564170 17699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:14.566422 17699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "5ddcd8f5f4564efeaf31983c4b575094"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "05abc76b0f9038864efbc02dd3c286bd"
server_key_iv: "f8af24da23c78e9e8804d1c63b6de2a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.567025 17699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "5ddcd8f5f4564efeaf31983c4b575094"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "05abc76b0f9038864efbc02dd3c286bd"
server_key_iv: "f8af24da23c78e9e8804d1c63b6de2a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.571112 17699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:14.573911 17714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:14.575313 17699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:14.575484 17699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "5ddcd8f5f4564efeaf31983c4b575094"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "05abc76b0f9038864efbc02dd3c286bd"
server_key_iv: "f8af24da23c78e9e8804d1c63b6de2a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.575616 17699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:14.592895 17699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:14.596305 17699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:14.596556 17699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:14.604035 17699 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41005
I20260620 21:55:14.604055 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41005 every 8 connection(s)
I20260620 21:55:14.605013 17699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:14.607807 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:14.610318 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 17699
I20260620 21:55:14.610486 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:14.611776 17679 external_mini_cluster.cc:1428] Setting key 2f81ed4125ba12ac64d1ea07f9e8ac97
I20260620 21:55:14.613323 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094: Bootstrap starting.
I20260620 21:55:14.616091 17767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:14.616825 17767 log.cc:824] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:14.618709 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094: No bootstrap required, opened a new log
I20260620 21:55:14.622448 17767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } }
I20260620 21:55:14.622884 17767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:14.623080 17767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddcd8f5f4564efeaf31983c4b575094, State: Initialized, Role: FOLLOWER
I20260620 21:55:14.623545 17767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } }
I20260620 21:55:14.623708 17767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:14.623806 17767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:14.623943 17767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:14.625072 17767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } }
I20260620 21:55:14.625451 17767 leader_election.cc:302] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ddcd8f5f4564efeaf31983c4b575094; no voters:
I20260620 21:55:14.625800 17767 leader_election.cc:288] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:14.625918 17772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:14.626128 17772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [term 1 LEADER]: Becoming Leader. State: Replica: 5ddcd8f5f4564efeaf31983c4b575094, State: Running, Role: LEADER
I20260620 21:55:14.626436 17772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } }
I20260620 21:55:14.627249 17767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:14.628222 17773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ddcd8f5f4564efeaf31983c4b575094. Latest consensus state: current_term: 1 leader_uuid: "5ddcd8f5f4564efeaf31983c4b575094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } } }
I20260620 21:55:14.628366 17773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:14.628249 17774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ddcd8f5f4564efeaf31983c4b575094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ddcd8f5f4564efeaf31983c4b575094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41005 } } }
I20260620 21:55:14.628625 17774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:14.628983 17781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 21:55:14 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:14.632212 17781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:14.637466 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:14.614857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.17.67.254:41005)
0620 21:55:14.615282 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:14.615298 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:14.616434 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 21:55:14.619813 (+ 3379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:14.619828 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:14.620123 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:14.620453 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:14.622643 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.623955 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.625152 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.625835 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.626065 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:14.631392 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:14.631401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:14.631420 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:14.631465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:14.633543 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:14.634097 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:14.634103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:14.634109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:14.634258 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:14.634701 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:14.634704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:14.634706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:14.635191 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:14.635383 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:14.636080 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 21:55:14.636372 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":145,"threads_started":1}
I20260620 21:55:14.642997 17781 catalog_manager.cc:1347] Generated new cluster ID: 310e4d70ca324191bb4ddacb7c0d3a58
I20260620 21:55:14.643072 17781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:14.653244 17781 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:14.654247 17781 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:14.660293 17781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ddcd8f5f4564efeaf31983c4b575094: Generated new TSK 0
I20260620 21:55:14.660982 17781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:14.697044 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:14.821285 17793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:14.821652 17793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:14.821753 17793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:14.825814 17793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:14.825943 17793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:14.826040 17793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:14.831805 17793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:14.832983 17793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:14.834084 17793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:14.843982 17799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:14.844010 17798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:14.843988 17801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:14.844452 17793 server_base.cc:1034] running on GCE node
I20260620 21:55:14.844848 17793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:14.845454 17793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:14.846622 17793 hybrid_clock.cc:648] HybridClock initialized: now 1781992514846604 us; error 30 us; skew 500 ppm
Jun 20 21:55:14 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992514, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:14.849380 17793 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:14.850370 17793 webserver.cc:460] Webserver started at http://127.17.67.193:43107/ using document root <none> and password file <none>
I20260620 21:55:14.850946 17793 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:14.851039 17793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:14.851295 17793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:14.853394 17793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "f6cd5b46405c45759a2d7b8bdd2078df"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "2eb72930ce075e82d9ee7bb07591aec4"
server_key_iv: "13e389ba4d4a0b367d8700ed848a3c94"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.854040 17793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "f6cd5b46405c45759a2d7b8bdd2078df"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "2eb72930ce075e82d9ee7bb07591aec4"
server_key_iv: "13e389ba4d4a0b367d8700ed848a3c94"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.857955 17793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:14.860380 17808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:14.861775 17793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:14.861913 17793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "f6cd5b46405c45759a2d7b8bdd2078df"
format_stamp: "Formatted at 2026-06-20 21:55:14 on dist-test-slave-cx8d"
server_key: "2eb72930ce075e82d9ee7bb07591aec4"
server_key_iv: "13e389ba4d4a0b367d8700ed848a3c94"
server_key_version: "encryptionkey@0"
I20260620 21:55:14.862035 17793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:14.881493 17793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:14.885085 17793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:14.885306 17793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:14.886016 17793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:14.886884 17793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:14.886947 17793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:14.887017 17793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:14.887049 17793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:14.895514 17793 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:41387
I20260620 21:55:14.895530 17921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:41387 every 8 connection(s)
I20260620 21:55:14.896402 17793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:14.904394 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 17793
I20260620 21:55:14.904538 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:14.904830 17679 external_mini_cluster.cc:1428] Setting key 049d031ae42d74a8f3c4519a5fbb84ee
Jun 20 21:55:14 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992514, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:14.911098 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:14.898291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:58293 (local address 127.17.67.254:41005)
0620 21:55:14.898466 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:14.898470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:14.899580 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 21:55:14.900491 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:14.900494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:14.900545 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:14.900629 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:14.901968 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.902546 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.903670 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.903905 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.903993 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:14.906863 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:14.906867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:14.906870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:14.906901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:14.908605 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:14.909190 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:14.909195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:14.909199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:14.909302 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:14.909702 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:14.909709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:14.909713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:14.909985 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:14.910091 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:14.910689 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 21:55:14.910841 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 21:55:14.911875 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:14.898668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41005 (local address 127.17.67.193:58293)
0620 21:55:14.899392 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:14.899448 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:14.900242 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:14.900811 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:14.900818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:14.901196 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:14.901813 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:14.901830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.902661 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.902664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:14.903556 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:14.903565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:14.904042 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:14.904045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:14.904183 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:14.904825 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:14.904842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:14.906698 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:14.908783 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:14.908788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:14.908801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:14.909083 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:14.909401 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:14.909404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:14.909405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:14.909550 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:14.910111 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:14.910117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:14.910413 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 21:55:14.910577 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:55:14.910928 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":109,"threads_started":1}
I20260620 21:55:14.913462 17922 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41005
I20260620 21:55:14.913739 17922 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:14.914248 17922 heartbeater.cc:511] Master 127.17.67.254:41005 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:14.915890 17731 ts_manager.cc:194] Registered new tserver with Master: f6cd5b46405c45759a2d7b8bdd2078df (127.17.67.193:41387)
I20260620 21:55:14.917101 17731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:58293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:14.932647 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:15.050753 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:15.051066 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:15.051147 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:15.055233 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:15.055341 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:15.055441 17927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:15.061343 17927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:15.062635 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:15.063781 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:15.074769 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:15.074725 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:15.074723 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:15.075099 17927 server_base.cc:1034] running on GCE node
I20260620 21:55:15.075551 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:15.076224 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:15.077467 17927 hybrid_clock.cc:648] HybridClock initialized: now 1781992515077422 us; error 58 us; skew 500 ppm
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:15.080545 17927 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:15.081610 17927 webserver.cc:460] Webserver started at http://127.17.67.194:37217/ using document root <none> and password file <none>
I20260620 21:55:15.082231 17927 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:15.082327 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:15.082551 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:15.084679 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "3fd92e7d19304d568cb578aed4ec0049"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "24697e2720b67c04a64dbe93ff4635a4"
server_key_iv: "9b89d0537b2f9744852ca05f0facbd24"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.085289 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "3fd92e7d19304d568cb578aed4ec0049"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "24697e2720b67c04a64dbe93ff4635a4"
server_key_iv: "9b89d0537b2f9744852ca05f0facbd24"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.089411 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:15.092451 17942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.093955 17927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:15.094067 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "3fd92e7d19304d568cb578aed4ec0049"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "24697e2720b67c04a64dbe93ff4635a4"
server_key_iv: "9b89d0537b2f9744852ca05f0facbd24"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.094172 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:15.108129 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:15.111581 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:15.111740 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:15.112282 17927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:15.113150 17927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:15.113224 17927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.113297 17927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:15.113358 17927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.122227 17927 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:38421
I20260620 21:55:15.122345 18055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:38421 every 8 connection(s)
I20260620 21:55:15.123154 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:15.129465 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 17927
I20260620 21:55:15.129616 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:15.129899 17679 external_mini_cluster.cc:1428] Setting key 0e43540d0a9c562e8c6794b9d56c1f8e
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:15.137617 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.125006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:38253 (local address 127.17.67.254:41005)
0620 21:55:15.125133 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:15.125137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:15.126525 (+ 1388us) server_negotiation.cc:407] Connection header received
0620 21:55:15.127433 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:15.127436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:15.127482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:15.127557 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:15.128803 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.129487 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.130756 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.130980 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.131101 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.133871 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:15.133880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:15.133883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:15.133906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:15.135394 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.135903 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.135907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.135909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.135991 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.136327 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.136332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.136334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.136521 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:15.136629 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:15.137234 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 21:55:15.137396 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:55:15.138435 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.125452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41005 (local address 127.17.67.194:38253)
0620 21:55:15.126370 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:15.126413 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:15.127233 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:15.127721 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:15.127765 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:15.128205 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:15.128675 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:15.128688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.129645 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.129649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:15.130629 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:15.130640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.131168 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.131172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:15.131412 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.132041 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:15.132058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:15.133704 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:15.135506 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:15.135511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:15.135527 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:15.135791 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:15.136085 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:15.136089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:15.136091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:15.136233 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:15.136626 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:15.136634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:15.136932 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 21:55:15.137153 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 21:55:15.137532 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":583,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:15.139736 18056 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41005
I20260620 21:55:15.140013 18056 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:15.140539 18056 heartbeater.cc:511] Master 127.17.67.254:41005 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:55:15.141803 17731 ts_manager.cc:194] Registered new tserver with Master: 3fd92e7d19304d568cb578aed4ec0049 (127.17.67.194:38421)
I20260620 21:55:15.142414 17731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:38253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:15.157063 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:15.268891 18061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:15.269157 18061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:15.269201 18061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:15.273190 18061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:15.273242 18061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:15.273300 18061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:15.278445 18061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:15.279556 18061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:15.280607 18061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:15.290381 18067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:15.290500 18061 server_base.cc:1034] running on GCE node
W20260620 21:55:15.290426 18069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:15.290684 18066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:15.291141 18061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:15.291782 18061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:15.292994 18061 hybrid_clock.cc:648] HybridClock initialized: now 1781992515292967 us; error 48 us; skew 500 ppm
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:15.295943 18061 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:15.296898 18061 webserver.cc:460] Webserver started at http://127.17.67.195:45881/ using document root <none> and password file <none>
I20260620 21:55:15.297619 18061 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:15.297719 18061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:15.297957 18061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:15.300153 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "e72bb980983d4fb8ab4b2e1c684717b1"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "3f9f7ab36920e967cf8a57bac29aa136"
server_key_iv: "3573d08f12b64b2150619f8b53c2b621"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.300734 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "e72bb980983d4fb8ab4b2e1c684717b1"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "3f9f7ab36920e967cf8a57bac29aa136"
server_key_iv: "3573d08f12b64b2150619f8b53c2b621"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.304662 18061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 21:55:15.307188 18076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.308511 18061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:15.308655 18061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "e72bb980983d4fb8ab4b2e1c684717b1"
format_stamp: "Formatted at 2026-06-20 21:55:15 on dist-test-slave-cx8d"
server_key: "3f9f7ab36920e967cf8a57bac29aa136"
server_key_iv: "3573d08f12b64b2150619f8b53c2b621"
server_key_version: "encryptionkey@0"
I20260620 21:55:15.308782 18061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:15.322396 18061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:15.325213 18061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:15.325448 18061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:15.326465 18061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:15.327267 18061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:15.327346 18061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.327417 18061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:15.327461 18061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:15.336406 18061 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33877
I20260620 21:55:15.336426 18189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33877 every 8 connection(s)
I20260620 21:55:15.337267 18061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:15.343384 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18061
I20260620 21:55:15.343528 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:15.343786 17679 external_mini_cluster.cc:1428] Setting key 15b55099430ac34de5a07d90e8b08b1c
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:15.351048 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.339128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:45435 (local address 127.17.67.254:41005)
0620 21:55:15.339277 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:15.339280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:15.340083 (+ 803us) server_negotiation.cc:407] Connection header received
0620 21:55:15.341428 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:15.341432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:15.341487 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:15.341645 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:15.342798 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.343359 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.344699 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.344925 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.345033 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.347526 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:15.347529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:15.347532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:15.347563 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:15.348826 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.349456 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.349460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.349462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.349545 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.349927 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.349930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.349932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.350114 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:15.350220 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:15.350736 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 21:55:15.350866 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:55:15.351962 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.339501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41005 (local address 127.17.67.195:45435)
0620 21:55:15.339928 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:15.339969 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:15.341222 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:15.341815 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:15.341822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:15.342170 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:15.342644 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:15.342657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.343515 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.343521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:15.344574 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:15.344586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.345090 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.345094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:15.345262 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.345853 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:15.345868 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:15.347325 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:15.348976 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:15.348981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:15.349015 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:15.349344 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:15.349662 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:15.349666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:15.349668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:15.349814 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:15.350219 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:15.350225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:15.350484 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 21:55:15.350673 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 21:55:15.351009 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260620 21:55:15.353164 18190 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41005
I20260620 21:55:15.353415 18190 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:15.354115 18190 heartbeater.cc:511] Master 127.17.67.254:41005 requested a full tablet report, sending...
I20260620 21:55:15.355193 17731 ts_manager.cc:194] Registered new tserver with Master: e72bb980983d4fb8ab4b2e1c684717b1 (127.17.67.195:33877)
I20260620 21:55:15.355811 17731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:45435
I20260620 21:55:15.358117 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:15.385757 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.375277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.17.67.193:41387)
0620 21:55:15.375634 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:15.375648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:15.375671 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:55:15.375805 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:15.375811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:15.375997 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:15.376120 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:15.376805 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.377641 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.378485 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.378866 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.378981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.381747 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:15.381756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:15.381768 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:15.381795 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:15.383810 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.384375 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.384383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.384387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.384501 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.384761 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.384763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.384764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.385005 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:15.385176 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:15.385316 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 21:55:15.385442 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":154,"threads_started":1}
W20260620 21:55:15.386713 17836 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59002
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[17686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:15.413911 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:15.404545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.17.67.193:41387)
0620 21:55:15.404697 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:15.404701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:15.404842 (+ 141us) server_negotiation.cc:407] Connection header received
0620 21:55:15.404955 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:15.404958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:15.405003 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:15.405088 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:15.406089 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.407080 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.407936 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:15.408184 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:15.408315 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:15.410877 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:15.410880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:15.410882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:15.410905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:15.412203 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.412675 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.412682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.412686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.412767 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:15.413057 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:15.413060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:15.413061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:15.413223 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:15.413364 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:15.413623 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:55:15.413738 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:55:15.414914 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 17793
I20260620 21:55:15.421703 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 17927
I20260620 21:55:15.426977 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18061
I20260620 21:55:15.432317 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 17699
2026-06-20T21:55:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2182 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[18204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[18204](info): set up 2 sockets
Jun 20 21:55:15 dist-test-slave-cx8d krb5kdc[18204](info): commencing operation
krb5kdc: starting...
W20260620 21:55:16.477198 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:16 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:16.616945 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43207
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:43207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:16.725329 18217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:16.725656 18217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:16.725764 18217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:16.729725 18217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:16.729810 18217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:16.729846 18217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:16.729884 18217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:16.729916 18217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:16.735018 18217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:43207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:16.736022 18217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:16.737017 18217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:16.745772 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:16.745772 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:16.745851 18217 server_base.cc:1034] running on GCE node
W20260620 21:55:16.745801 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:16.746618 18217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:16.747793 18217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:16.748996 18217 hybrid_clock.cc:648] HybridClock initialized: now 1781992516748969 us; error 55 us; skew 500 ppm
Jun 20 21:55:16 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992516, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:16.751958 18217 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:16.752995 18217 webserver.cc:460] Webserver started at http://127.17.67.254:37481/ using document root <none> and password file <none>
I20260620 21:55:16.753695 18217 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:16.753793 18217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:16.754019 18217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:16.756274 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "1f3e0eb211f14b37973281c727f5a9a4"
format_stamp: "Formatted at 2026-06-20 21:55:16 on dist-test-slave-cx8d"
server_key: "11c462ab6b85936e10dc19d457029b5b"
server_key_iv: "ecdbdf45cb5681313c2f8a4efe6b4924"
server_key_version: "encryptionkey@0"
I20260620 21:55:16.756929 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "1f3e0eb211f14b37973281c727f5a9a4"
format_stamp: "Formatted at 2026-06-20 21:55:16 on dist-test-slave-cx8d"
server_key: "11c462ab6b85936e10dc19d457029b5b"
server_key_iv: "ecdbdf45cb5681313c2f8a4efe6b4924"
server_key_version: "encryptionkey@0"
I20260620 21:55:16.760950 18217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:16.763566 18232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:16.764868 18217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:16.765034 18217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "1f3e0eb211f14b37973281c727f5a9a4"
format_stamp: "Formatted at 2026-06-20 21:55:16 on dist-test-slave-cx8d"
server_key: "11c462ab6b85936e10dc19d457029b5b"
server_key_iv: "ecdbdf45cb5681313c2f8a4efe6b4924"
server_key_version: "encryptionkey@0"
I20260620 21:55:16.765158 18217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:16.788367 18217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:16.791687 18217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:16.791913 18217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:16.799431 18217 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:43207
I20260620 21:55:16.799453 18284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:43207 every 8 connection(s)
I20260620 21:55:16.800400 18217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:16.803122 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18217
I20260620 21:55:16.803211 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:16.803435 18285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:16.803501 17679 external_mini_cluster.cc:1428] Setting key 3bee488141afb9443af633fe7d28b171
I20260620 21:55:16.811350 18285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4: Bootstrap starting.
Jun 20 21:55:16 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:16.814225 18285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:16.815182 18285 log.cc:824] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:16.817531 18285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4: No bootstrap required, opened a new log
I20260620 21:55:16.820158 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:16.804956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.17.67.254:43207)
0620 21:55:16.805393 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:16.805416 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:16.805462 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:55:16.806470 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:16.806496 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:16.806957 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:16.807369 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:16.808042 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:16.809726 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:16.810741 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:16.811505 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:16.811673 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:16.814884 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:16.814891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:16.814907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:16.814940 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:16.816821 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:16.817366 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:16.817371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:16.817377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:16.817501 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:16.817995 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:16.817997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:16.817999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:16.818419 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:16.818568 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:16.818968 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 21:55:16.819332 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260620 21:55:16.821913 18285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } }
I20260620 21:55:16.822184 18285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:16.822259 18285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3e0eb211f14b37973281c727f5a9a4, State: Initialized, Role: FOLLOWER
I20260620 21:55:16.822722 18285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } }
I20260620 21:55:16.822850 18285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:16.822896 18285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:16.822968 18285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:16.823835 18285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } }
I20260620 21:55:16.824200 18285 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3e0eb211f14b37973281c727f5a9a4; no voters:
I20260620 21:55:16.824625 18285 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:16.824745 18290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:16.824978 18290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1f3e0eb211f14b37973281c727f5a9a4, State: Running, Role: LEADER
I20260620 21:55:16.825346 18290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } }
I20260620 21:55:16.825902 18285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:16.827037 18292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3e0eb211f14b37973281c727f5a9a4. Latest consensus state: current_term: 1 leader_uuid: "1f3e0eb211f14b37973281c727f5a9a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } } }
I20260620 21:55:16.827167 18292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:16.827600 18299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:16.827564 18291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3e0eb211f14b37973281c727f5a9a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3e0eb211f14b37973281c727f5a9a4" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43207 } } }
I20260620 21:55:16.827896 18291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:16.831311 18299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:16.837257 18299 catalog_manager.cc:1347] Generated new cluster ID: 6a90558efc8c489a94d147dcd33084d0
I20260620 21:55:16.837338 18299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:16.844841 18299 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:16.846073 18299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:16.853298 18299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f3e0eb211f14b37973281c727f5a9a4: Generated new TSK 0
I20260620 21:55:16.854079 18299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:16.889318 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:17.007316 18311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:17.007561 18311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:17.007606 18311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:17.011583 18311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:17.011641 18311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:17.011698 18311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:17.016878 18311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:17.017848 18311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:17.018800 18311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:17.028529 18316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:17.028687 18317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:17.028548 18319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:17.029291 18311 server_base.cc:1034] running on GCE node
I20260620 21:55:17.029812 18311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:17.030611 18311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:17.031801 18311 hybrid_clock.cc:648] HybridClock initialized: now 1781992517031767 us; error 45 us; skew 500 ppm
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:17.034878 18311 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:17.035708 18311 webserver.cc:460] Webserver started at http://127.17.67.193:33055/ using document root <none> and password file <none>
I20260620 21:55:17.036278 18311 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:17.036340 18311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:17.036506 18311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:17.038713 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "2e6539c18d164316bb9965cf82a53e6e"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "d3cfed5423466858fca47b2d3e7383e0"
server_key_iv: "5be362c3b6d6c444485caa4b8159f4d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.039263 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "2e6539c18d164316bb9965cf82a53e6e"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "d3cfed5423466858fca47b2d3e7383e0"
server_key_iv: "5be362c3b6d6c444485caa4b8159f4d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.043129 18311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:17.045852 18326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.047169 18311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:17.047335 18311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "2e6539c18d164316bb9965cf82a53e6e"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "d3cfed5423466858fca47b2d3e7383e0"
server_key_iv: "5be362c3b6d6c444485caa4b8159f4d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.047448 18311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:17.066054 18311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:17.069840 18311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:17.070055 18311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:17.070721 18311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:17.071595 18311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:17.071645 18311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.071688 18311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:17.071749 18311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.081547 18311 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:45977
I20260620 21:55:17.081633 18439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:45977 every 8 connection(s)
I20260620 21:55:17.082509 18311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:17.086915 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18311
I20260620 21:55:17.087031 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:17.087297 17679 external_mini_cluster.cc:1428] Setting key f9e5c77e096c4272d68e51071459a9ca
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:17.097433 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.084321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:45509 (local address 127.17.67.254:43207)
0620 21:55:17.084463 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.084467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.085304 (+ 837us) server_negotiation.cc:407] Connection header received
0620 21:55:17.086434 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.086438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.086496 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.086613 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.087999 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.088588 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.089906 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.090373 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.090500 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.093098 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.093106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.093110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.093145 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.095191 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.095686 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.095691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.095693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.095805 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.096143 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.096147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.096149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.096431 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.096546 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.097047 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.097197 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:17.098102 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.084688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:43207 (local address 127.17.67.193:45509)
0620 21:55:17.085136 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.085180 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.086216 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.086832 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.086840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.087289 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:17.087843 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.087858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.088745 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.088749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.089775 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.089785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.090533 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.090536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.090681 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.091427 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:17.091449 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:17.092917 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:17.095312 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.095317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.095329 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.095573 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.095921 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.095923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.095924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.096047 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.096551 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:17.096555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:17.096771 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.096946 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.097183 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260620 21:55:17.099428 18440 heartbeater.cc:347] Connected to a master server at 127.17.67.254:43207
I20260620 21:55:17.099694 18440 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:17.100224 18440 heartbeater.cc:511] Master 127.17.67.254:43207 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:55:17.101958 18249 ts_manager.cc:194] Registered new tserver with Master: 2e6539c18d164316bb9965cf82a53e6e (127.17.67.193:45977)
I20260620 21:55:17.103045 18249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:45509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:17.116623 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:17.227685 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:17.227955 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:17.228037 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:17.232113 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:17.232216 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:17.232311 18445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:17.237602 18445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:17.238684 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:17.239723 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:17.249419 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:17.249495 18445 server_base.cc:1034] running on GCE node
W20260620 21:55:17.249419 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:17.249433 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:17.250087 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:17.250813 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:17.252010 18445 hybrid_clock.cc:648] HybridClock initialized: now 1781992517251995 us; error 27 us; skew 500 ppm
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:17.254833 18445 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:17.255757 18445 webserver.cc:460] Webserver started at http://127.17.67.194:37701/ using document root <none> and password file <none>
I20260620 21:55:17.256327 18445 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:17.256418 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:17.256647 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:17.258744 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "0656a67256914e20a2c75815e80e3800"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "29ba3163f15c80fc77ebcf602f2bb210"
server_key_iv: "1f43501c584058b5a06d6eed4bf28748"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.259316 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "0656a67256914e20a2c75815e80e3800"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "29ba3163f15c80fc77ebcf602f2bb210"
server_key_iv: "1f43501c584058b5a06d6eed4bf28748"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.263098 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:17.265559 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.266976 18445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:17.267076 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "0656a67256914e20a2c75815e80e3800"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "29ba3163f15c80fc77ebcf602f2bb210"
server_key_iv: "1f43501c584058b5a06d6eed4bf28748"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.267227 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:17.288209 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:17.291352 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:17.291559 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:17.292173 18445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:17.292989 18445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:17.293033 18445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.293100 18445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:17.293134 18445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.302150 18445 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40559
I20260620 21:55:17.302147 18573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40559 every 8 connection(s)
I20260620 21:55:17.303048 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:17.313179 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18445
I20260620 21:55:17.313304 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:17.313658 17679 external_mini_cluster.cc:1428] Setting key 03901b49db76aad65dc1e54a0501983a
I20260620 21:55:17.316466 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.304780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54115 (local address 127.17.67.254:43207)
0620 21:55:17.304917 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.304920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.305802 (+ 882us) server_negotiation.cc:407] Connection header received
0620 21:55:17.306699 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.306703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.306756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.306861 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.308046 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.308583 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.309765 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.309996 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.310106 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.312537 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.312540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.312542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.312564 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.314307 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.314783 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.314787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.314789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.314876 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.315258 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.315260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.315262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.315435 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.315514 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.316214 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.316353 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:17.317425 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.305172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:43207 (local address 127.17.67.194:54115)
0620 21:55:17.305661 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.305703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.306500 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.307066 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.307073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.307429 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:17.307900 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.307913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.308710 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.308720 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.309649 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.309658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.310143 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.310148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.310313 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.310887 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:17.310903 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:17.312379 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:17.314424 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.314429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.314441 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.314689 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.314975 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.314978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.314980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.315159 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.315543 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:17.315552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:17.315902 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.316118 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.316512 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260620 21:55:17.318601 18574 heartbeater.cc:347] Connected to a master server at 127.17.67.254:43207
I20260620 21:55:17.318835 18574 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:17.319330 18574 heartbeater.cc:511] Master 127.17.67.254:43207 requested a full tablet report, sending...
I20260620 21:55:17.320444 18249 ts_manager.cc:194] Registered new tserver with Master: 0656a67256914e20a2c75815e80e3800 (127.17.67.194:40559)
I20260620 21:55:17.320995 18249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:54115
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:17.338888 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:17.451699 18579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:17.451989 18579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:17.452080 18579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:17.456157 18579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:17.456244 18579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:17.456350 18579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:17.461658 18579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:43207
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:17.462734 18579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:17.463783 18579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:17.473827 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:17.473934 18579 server_base.cc:1034] running on GCE node
W20260620 21:55:17.473848 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:17.473829 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:17.474526 18579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:17.475172 18579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:17.476356 18579 hybrid_clock.cc:648] HybridClock initialized: now 1781992517476343 us; error 29 us; skew 500 ppm
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:17.479218 18579 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:17.480136 18579 webserver.cc:460] Webserver started at http://127.17.67.195:45325/ using document root <none> and password file <none>
I20260620 21:55:17.480975 18579 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:17.481056 18579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:17.481271 18579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:17.483448 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "c3dce2b6f24f420f90e4e9163d468f78"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "809147d90e2fa4fece1bfd5948896191"
server_key_iv: "b86011a6807002bf86042e61c2f3b65b"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.484042 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "c3dce2b6f24f420f90e4e9163d468f78"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "809147d90e2fa4fece1bfd5948896191"
server_key_iv: "b86011a6807002bf86042e61c2f3b65b"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.487917 18579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:17.490311 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.491541 18579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:17.491669 18579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "c3dce2b6f24f420f90e4e9163d468f78"
format_stamp: "Formatted at 2026-06-20 21:55:17 on dist-test-slave-cx8d"
server_key: "809147d90e2fa4fece1bfd5948896191"
server_key_iv: "b86011a6807002bf86042e61c2f3b65b"
server_key_version: "encryptionkey@0"
I20260620 21:55:17.491771 18579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:17.518644 18579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:17.521703 18579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:17.521893 18579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:17.522464 18579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:17.523243 18579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:17.523310 18579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.523386 18579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:17.523437 18579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:17.532325 18579 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:39841
I20260620 21:55:17.532375 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:39841 every 8 connection(s)
I20260620 21:55:17.533169 18579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:17.535143 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18579
I20260620 21:55:17.535267 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:17.535517 17679 external_mini_cluster.cc:1428] Setting key aabb6df324058ed4e431d77362a34bbb
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:17.546483 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.534893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:34881 (local address 127.17.67.254:43207)
0620 21:55:17.535056 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.535062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.536007 (+ 945us) server_negotiation.cc:407] Connection header received
0620 21:55:17.537056 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.537059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.537109 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.537188 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.538446 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.538976 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.540095 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.540319 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.540427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.542947 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.542950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.542952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.542974 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.544400 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.545004 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.545007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.545008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.545081 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.545456 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.545458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.545459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.545701 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.545780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.546245 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.546342 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:55:17.547340 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.535335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:43207 (local address 127.17.67.195:34881)
0620 21:55:17.535857 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.535910 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.536866 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.537363 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.537370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.537840 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:17.538311 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.538324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.539099 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.539102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.539973 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.539981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.540461 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.540465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.540631 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.541159 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:17.541207 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:17.542796 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:17.544548 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.544554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.544568 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.544899 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.545193 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:17.545195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:17.545197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:17.545349 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:17.545795 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:17.545801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:17.546044 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.546204 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.546494 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260620 21:55:17.548640 18708 heartbeater.cc:347] Connected to a master server at 127.17.67.254:43207
I20260620 21:55:17.548898 18708 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:17.549460 18708 heartbeater.cc:511] Master 127.17.67.254:43207 requested a full tablet report, sending...
I20260620 21:55:17.550611 18249 ts_manager.cc:194] Registered new tserver with Master: c3dce2b6f24f420f90e4e9163d468f78 (127.17.67.195:39841)
I20260620 21:55:17.551182 18249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:34881
I20260620 21:55:17.561558 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:55:17.573405 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.565319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.17.67.254:43207)
0620 21:55:17.565473 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.565477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.565757 (+ 280us) server_negotiation.cc:407] Connection header received
0620 21:55:17.565806 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.565809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.565853 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.565923 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.566643 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.567152 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.568101 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.568433 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.568564 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.570154 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.570157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.570159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.570183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.571727 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.572126 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.572128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.572130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.572211 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.572552 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.572554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.572556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.572788 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.572920 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.573184 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.573282 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:55:17.578464 18249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:17.581707 18249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:17.596668 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.590180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:39841 (local address 127.0.0.1:35108)
0620 21:55:17.590649 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.590695 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.590824 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.592030 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.592036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.592073 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:17.592289 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.592297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.593488 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.593494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.594759 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.594769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.596058 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.596062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.596192 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.596235 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.596358 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.596495 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":37,"spinlock_wait_cycles":7936,"thread_start_us":81,"threads_started":1}
I20260620 21:55:17.597404 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.590360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.17.67.195:39841)
0620 21:55:17.591580 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.591591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.591610 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:55:17.591675 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.591680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.591855 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.592439 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:17.592592 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.593354 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.594910 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.595884 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.596051 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.596226 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.597096 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.597230 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":143,"threads_started":1}
I20260620 21:55:17.597618 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.590924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:45977 (local address 127.0.0.1:36278)
0620 21:55:17.591238 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.591252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.591337 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.592502 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.592504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.592520 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:17.592618 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.592627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.594078 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.594081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.595566 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.595577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.597083 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.597087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.597197 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.597227 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.597311 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.597380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":73,"thread_start_us":102,"threads_started":1}
I20260620 21:55:17.597661 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.590361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40559 (local address 127.0.0.1:52938)
0620 21:55:17.591212 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.591228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.591329 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.592032 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.592037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.592072 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:17.592369 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.592378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.594062 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.594068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.595782 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.595794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.597133 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.597136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.597227 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.597246 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.597358 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.597435 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":223,"spinlock_wait_cycles":6656,"thread_start_us":115,"threads_started":1}
I20260620 21:55:17.598109 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.591420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.17.67.193:45977)
0620 21:55:17.591963 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.591980 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.592004 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:55:17.592082 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.592088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.592342 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.592539 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:17.592719 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.593941 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.595745 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.596739 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.596858 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.597014 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.597371 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.597961 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":90,"threads_started":1}
I20260620 21:55:17.598464 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.590971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.17.67.194:40559)
0620 21:55:17.591336 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.591349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.591384 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:55:17.591467 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.591473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.591688 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.591852 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:17.592496 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.593881 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.595944 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.596801 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.596921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.597062 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.597938 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.598300 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":132,"threads_started":1}
I20260620 21:55:17.600895 18508 tablet_service.cc:1467] Processing CreateTablet for tablet 36bd72f35b064e98a1f1c0738bd3bb9d (DEFAULT_TABLE table=test-table [id=e259523d97ff4e7297304f46a4e45430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:17.600744 18642 tablet_service.cc:1467] Processing CreateTablet for tablet 36bd72f35b064e98a1f1c0738bd3bb9d (DEFAULT_TABLE table=test-table [id=e259523d97ff4e7297304f46a4e45430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:17.601300 18374 tablet_service.cc:1467] Processing CreateTablet for tablet 36bd72f35b064e98a1f1c0738bd3bb9d (DEFAULT_TABLE table=test-table [id=e259523d97ff4e7297304f46a4e45430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:17.602005 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bd72f35b064e98a1f1c0738bd3bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:17.602291 18374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bd72f35b064e98a1f1c0738bd3bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:17.602284 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36bd72f35b064e98a1f1c0738bd3bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:17.608722 18725 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: Bootstrap starting.
I20260620 21:55:17.610222 18726 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Bootstrap starting.
I20260620 21:55:17.611341 18727 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: Bootstrap starting.
I20260620 21:55:17.611905 18725 tablet_bootstrap.cc:654] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:17.612974 18726 tablet_bootstrap.cc:654] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:17.613396 18725 log.cc:824] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:17.613698 18727 tablet_bootstrap.cc:654] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:17.613899 18726 log.cc:824] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:17.614555 18727 log.cc:824] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:17.615999 18725 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: No bootstrap required, opened a new log
I20260620 21:55:17.616000 18726 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: No bootstrap required, opened a new log
I20260620 21:55:17.616197 18725 ts_tablet_manager.cc:1397] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 21:55:17.616212 18726 ts_tablet_manager.cc:1397] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:17.616441 18727 tablet_bootstrap.cc:492] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: No bootstrap required, opened a new log
I20260620 21:55:17.616657 18727 ts_tablet_manager.cc:1397] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:17.620587 18727 raft_consensus.cc:348] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.620141 18726 raft_consensus.cc:348] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.620141 18725 raft_consensus.cc:348] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.620836 18727 raft_consensus.cc:374] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:17.620836 18726 raft_consensus.cc:374] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:17.620836 18725 raft_consensus.cc:374] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:17.620898 18727 raft_consensus.cc:729] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6539c18d164316bb9965cf82a53e6e, State: Initialized, Role: FOLLOWER
I20260620 21:55:17.620898 18725 raft_consensus.cc:729] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0656a67256914e20a2c75815e80e3800, State: Initialized, Role: FOLLOWER
I20260620 21:55:17.620898 18726 raft_consensus.cc:729] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dce2b6f24f420f90e4e9163d468f78, State: Initialized, Role: FOLLOWER
I20260620 21:55:17.621366 18725 consensus_queue.cc:260] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.621366 18726 consensus_queue.cc:260] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.621388 18727 consensus_queue.cc:260] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.622279 18708 heartbeater.cc:503] Master 127.17.67.254:43207 was elected leader, sending a full tablet report...
I20260620 21:55:17.622401 18726 ts_tablet_manager.cc:1428] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:55:17.622401 18725 ts_tablet_manager.cc:1428] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 21:55:17.622426 18727 ts_tablet_manager.cc:1428] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:17.622743 18440 heartbeater.cc:503] Master 127.17.67.254:43207 was elected leader, sending a full tablet report...
I20260620 21:55:17.622741 18574 heartbeater.cc:503] Master 127.17.67.254:43207 was elected leader, sending a full tablet report...
I20260620 21:55:17.689833 18732 raft_consensus.cc:482] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:17.690066 18732 raft_consensus.cc:504] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.691298 18732 leader_election.cc:288] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0656a67256914e20a2c75815e80e3800 (127.17.67.194:40559), 2e6539c18d164316bb9965cf82a53e6e (127.17.67.193:45977)
I20260620 21:55:17.695278 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.691495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40559 (local address 127.17.67.195:33983)
0620 21:55:17.691612 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.691630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.691725 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.692006 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.692008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.692024 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:17.692187 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.692195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.693127 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.693130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.694016 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.694025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.694848 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.694851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.694954 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.694981 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.695068 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.695140 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 21:55:17.695360 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.691613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:33983 (local address 127.17.67.194:40559)
0620 21:55:17.691751 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.691755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.691772 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:55:17.691847 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.691850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.691903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.691986 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:17.692333 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.693019 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.694132 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.694736 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.694830 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.694872 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.695136 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.695206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:17.695554 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.691577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:45977 (local address 127.17.67.195:60789)
0620 21:55:17.692000 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:17.692013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:17.692113 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:17.692429 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:17.692431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:17.692450 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:17.692573 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.692580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.693397 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.693400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.694386 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:17.694396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.695145 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.695148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:17.695243 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.695265 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:17.695344 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:55:17.695415 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":129,"threads_started":1}
I20260620 21:55:17.695605 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.691790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:60789 (local address 127.17.67.193:45977)
0620 21:55:17.691898 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.691902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.692097 (+ 195us) server_negotiation.cc:407] Connection header received
0620 21:55:17.692211 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.692216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.692279 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.692380 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:17.692706 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.693283 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.694505 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.695017 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.695125 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.695166 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.695384 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.695458 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 21:55:17.696220 18394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bd72f35b064e98a1f1c0738bd3bb9d" candidate_uuid: "c3dce2b6f24f420f90e4e9163d468f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6539c18d164316bb9965cf82a53e6e" is_pre_election: true
I20260620 21:55:17.696266 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bd72f35b064e98a1f1c0738bd3bb9d" candidate_uuid: "c3dce2b6f24f420f90e4e9163d468f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0656a67256914e20a2c75815e80e3800" is_pre_election: true
I20260620 21:55:17.696517 18394 raft_consensus.cc:2454] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dce2b6f24f420f90e4e9163d468f78 in term 0.
I20260620 21:55:17.696533 18528 raft_consensus.cc:2454] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dce2b6f24f420f90e4e9163d468f78 in term 0.
I20260620 21:55:17.696973 18596 leader_election.cc:302] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0656a67256914e20a2c75815e80e3800, c3dce2b6f24f420f90e4e9163d468f78; no voters:
I20260620 21:55:17.697247 18732 raft_consensus.cc:2781] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:17.697337 18732 raft_consensus.cc:482] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:17.697381 18732 raft_consensus.cc:3037] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:17.698402 18732 raft_consensus.cc:504] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.698737 18732 leader_election.cc:288] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [CANDIDATE]: Term 1 election: Requested vote from peers 0656a67256914e20a2c75815e80e3800 (127.17.67.194:40559), 2e6539c18d164316bb9965cf82a53e6e (127.17.67.193:45977)
I20260620 21:55:17.699035 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bd72f35b064e98a1f1c0738bd3bb9d" candidate_uuid: "c3dce2b6f24f420f90e4e9163d468f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0656a67256914e20a2c75815e80e3800"
I20260620 21:55:17.699100 18394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36bd72f35b064e98a1f1c0738bd3bb9d" candidate_uuid: "c3dce2b6f24f420f90e4e9163d468f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6539c18d164316bb9965cf82a53e6e"
I20260620 21:55:17.699188 18394 raft_consensus.cc:3037] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:17.699174 18528 raft_consensus.cc:3037] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:17.700130 18394 raft_consensus.cc:2454] T 36bd72f35b064e98a1f1c0738bd3bb9d P 2e6539c18d164316bb9965cf82a53e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dce2b6f24f420f90e4e9163d468f78 in term 1.
I20260620 21:55:17.700220 18528 raft_consensus.cc:2454] T 36bd72f35b064e98a1f1c0738bd3bb9d P 0656a67256914e20a2c75815e80e3800 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dce2b6f24f420f90e4e9163d468f78 in term 1.
I20260620 21:55:17.700433 18598 leader_election.cc:302] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6539c18d164316bb9965cf82a53e6e, c3dce2b6f24f420f90e4e9163d468f78; no voters:
I20260620 21:55:17.700604 18732 raft_consensus.cc:2781] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:17.700847 18732 raft_consensus.cc:686] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [term 1 LEADER]: Becoming Leader. State: Replica: c3dce2b6f24f420f90e4e9163d468f78, State: Running, Role: LEADER
I20260620 21:55:17.701197 18732 consensus_queue.cc:237] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } }
I20260620 21:55:17.704547 18249 catalog_manager.cc:5495] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dce2b6f24f420f90e4e9163d468f78 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "c3dce2b6f24f420f90e4e9163d468f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3dce2b6f24f420f90e4e9163d468f78" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 39841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0656a67256914e20a2c75815e80e3800" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6539c18d164316bb9965cf82a53e6e" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 45977 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:17.720752 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.712442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.17.67.193:45977)
0620 21:55:17.712549 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.712552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.712609 (+ 57us) server_negotiation.cc:407] Connection header received
0620 21:55:17.712713 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.712715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.712756 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.712820 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.713498 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.714086 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.714971 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.715202 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.715287 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.717844 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.717855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.717868 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.717890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.719336 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.719715 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.719720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.719721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.719795 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.720026 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.720029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.720030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.720254 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.720401 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.720542 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.720633 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:17.747131 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.738751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.17.67.193:45977)
0620 21:55:17.738885 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.738890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.739067 (+ 177us) server_negotiation.cc:407] Connection header received
0620 21:55:17.739246 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.739248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.739291 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.739398 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.740127 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.740640 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.741474 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.741770 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.741886 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.744207 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.744210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.744212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.744236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.745470 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.745912 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.745917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.745919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.746001 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.746266 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.746271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.746272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.746444 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.746578 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.746851 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.746969 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:17.772804 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:17.764250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.17.67.193:45977)
0620 21:55:17.764399 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:17.764403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:17.764578 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:55:17.764689 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:17.764692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:17.764734 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:17.764803 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:17.765532 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.766074 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.766901 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:17.767106 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:17.767183 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:17.769905 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:17.769909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:17.769910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:17.769933 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:17.771269 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.771659 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.771661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.771662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.771768 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:17.771997 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:17.771999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:17.772000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:17.772152 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:17.772252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:17.772523 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 21:55:17.772614 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 21:55:17.774184 18354 tablet_service.cc:2908] Rejecting scan request for tablet 36bd72f35b064e98a1f1c0738bd3bb9d: Uninitialized: clean time has not yet been initialized
I20260620 21:55:17.775986 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18311
I20260620 21:55:17.781819 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18445
W20260620 21:55:17.784056 18709 tablet.cc:2369] T 36bd72f35b064e98a1f1c0738bd3bb9d P c3dce2b6f24f420f90e4e9163d468f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:17.787446 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18579
I20260620 21:55:17.793229 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18217
2026-06-20T21:55:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2361 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18749](info): set up 2 sockets
Jun 20 21:55:17 dist-test-slave-cx8d krb5kdc[18749](info): commencing operation
krb5kdc: starting...
W20260620 21:55:18.829485 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:18 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:18.953094 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39009
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:19.063570 18762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:19.063879 18762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:19.063947 18762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:19.068068 18762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:19.068153 18762 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:55:19.068185 18762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:19.068226 18762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:19.068261 18762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:19.068315 18762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:19.073627 18762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:19.074668 18762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:19.075683 18762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:19.084458 18768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.084455 18770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.084743 18762 server_base.cc:1034] running on GCE node
W20260620 21:55:19.084856 18767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.085448 18762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:19.086648 18762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:19.087862 18762 hybrid_clock.cc:648] HybridClock initialized: now 1781992519087825 us; error 48 us; skew 500 ppm
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:19.094836 18762 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:19.095819 18762 webserver.cc:460] Webserver started at http://127.17.67.254:39179/ using document root <none> and password file <none>
I20260620 21:55:19.096333 18762 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:19.096390 18762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:19.096549 18762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:19.098666 18762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "4fa8c21447a948e6a7171151dacb3727"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "d92f428e8d49c6c170b03a465f303722"
server_key_iv: "36ad7f9100fc20c0325ed7ad8d968d2e"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.099186 18762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "4fa8c21447a948e6a7171151dacb3727"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "d92f428e8d49c6c170b03a465f303722"
server_key_iv: "36ad7f9100fc20c0325ed7ad8d968d2e"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.102751 18762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:55:19.105087 18777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.106376 18762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:55:19.106474 18762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "4fa8c21447a948e6a7171151dacb3727"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "d92f428e8d49c6c170b03a465f303722"
server_key_iv: "36ad7f9100fc20c0325ed7ad8d968d2e"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.106557 18762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:19.118146 18762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:19.121274 18762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:19.121419 18762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:19.129032 18762 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39009
I20260620 21:55:19.129048 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39009 every 8 connection(s)
I20260620 21:55:19.130020 18762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:19.132715 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:19.138048 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Bootstrap starting.
I20260620 21:55:19.139118 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18762
I20260620 21:55:19.139247 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:19.139513 17679 external_mini_cluster.cc:1428] Setting key f30568a4a763eceb5a9a106c751a1d08
I20260620 21:55:19.140837 18840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:19.141638 18840 log.cc:824] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:19.143528 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: No bootstrap required, opened a new log
I20260620 21:55:19.146864 18840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } }
I20260620 21:55:19.147109 18840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:19.147190 18840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa8c21447a948e6a7171151dacb3727, State: Initialized, Role: FOLLOWER
I20260620 21:55:19.147653 18840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } }
I20260620 21:55:19.147800 18840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:19.147886 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:19.147982 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:19.148905 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } }
I20260620 21:55:19.149281 18840 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa8c21447a948e6a7171151dacb3727; no voters:
I20260620 21:55:19.149667 18840 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:19.149760 18845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:19.149927 18845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa8c21447a948e6a7171151dacb3727, State: Running, Role: LEADER
I20260620 21:55:19.150193 18845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } }
I20260620 21:55:19.150939 18840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:19.151337 18846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa8c21447a948e6a7171151dacb3727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } } }
I20260620 21:55:19.151477 18846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:19.151347 18845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa8c21447a948e6a7171151dacb3727. Latest consensus state: current_term: 1 leader_uuid: "4fa8c21447a948e6a7171151dacb3727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa8c21447a948e6a7171151dacb3727" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39009 } } }
I20260620 21:55:19.151772 18845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:19.154549 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.140626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.17.67.254:39009)
0620 21:55:19.141113 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.141128 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.141163 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:55:19.141925 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.141944 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.142409 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.142751 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.143525 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.144527 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.145361 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.145968 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.146146 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.148834 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.148839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.148852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.148878 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.150875 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.151428 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.151434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.151440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.151574 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.151925 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.151929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.151931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.152437 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.152628 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.153519 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.153837 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":157,"threads_started":1}
W20260620 21:55:19.155984 18860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:19.156062 18860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:19.156149 18849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:19.156831 18849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:19.162755 18849 catalog_manager.cc:1347] Generated new cluster ID: 21a58313fb1f43b4bbff32bbd2bfae2a
I20260620 21:55:19.162829 18849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:19.189956 18849 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:19.190848 18849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:19.196635 18849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Generated new TSK 0
I20260620 21:55:19.197296 18849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:19.228682 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39009
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:19.337878 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:19.338121 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:19.338201 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:19.342129 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:19.342180 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:19.342253 18866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:19.342310 18866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:19.347471 18866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:19.348484 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:19.349499 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:19.359211 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.359269 18866 server_base.cc:1034] running on GCE node
W20260620 21:55:19.359254 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.359215 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.359930 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:19.360613 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:19.361764 18866 hybrid_clock.cc:648] HybridClock initialized: now 1781992519361743 us; error 31 us; skew 500 ppm
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:19.364557 18866 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:19.365401 18866 webserver.cc:460] Webserver started at http://127.17.67.193:44615/ using document root <none> and password file <none>
I20260620 21:55:19.366022 18866 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:19.366111 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:19.366341 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:19.368328 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "9247270bac0f4af4adc2de99ddf82773"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "fa82abcd8890a657e78e6a8bbe838d91"
server_key_iv: "7aab32d4b726645ec09a0481e2eaf4a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.368887 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "9247270bac0f4af4adc2de99ddf82773"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "fa82abcd8890a657e78e6a8bbe838d91"
server_key_iv: "7aab32d4b726645ec09a0481e2eaf4a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.372454 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:55:19.374946 18881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.376122 18866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:19.376250 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "9247270bac0f4af4adc2de99ddf82773"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "fa82abcd8890a657e78e6a8bbe838d91"
server_key_iv: "7aab32d4b726645ec09a0481e2eaf4a4"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.376360 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:19.395789 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:19.399066 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:19.399276 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:19.400753 18866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:19.400835 18866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.400908 18866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:19.400950 18866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:19.411985 18866 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:40897
I20260620 21:55:19.412185 18996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:40897 every 8 connection(s)
I20260620 21:55:19.412894 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:19.414680 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 18866
I20260620 21:55:19.414793 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:19.415102 17679 external_mini_cluster.cc:1428] Setting key d0a881e7a2ba8c7dcda440a194a9a7bb
I20260620 21:55:19.415406 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.401756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:39447 (local address 127.17.67.254:39009)
0620 21:55:19.401889 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.401892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.402920 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 21:55:19.404005 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.404009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.404064 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.404162 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.405874 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.406479 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.407717 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.408107 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.408223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.411475 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.411479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.411482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.411510 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.413190 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.413830 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.413834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.413836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.413927 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.414226 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.414229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.414230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.414430 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.414507 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.415113 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.415260 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:55:19.416374 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.402313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.193:39447)
0620 21:55:19.402777 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.402824 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.403733 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.404418 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.404430 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.404917 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.405653 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.405678 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.406609 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.406613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.407565 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.407574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.408339 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.408346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.408563 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.409209 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.409230 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.411282 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.413309 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.413315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.413329 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.413699 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.414014 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.414017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.414018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.414141 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.414534 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.414542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.414860 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.415060 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.415376 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260620 21:55:19.417549 18997 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39009
I20260620 21:55:19.417721 18997 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:19.418287 18997 heartbeater.cc:511] Master 127.17.67.254:39009 requested a full tablet report, sending...
I20260620 21:55:19.419771 18793 ts_manager.cc:194] Registered new tserver with Master: 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897)
I20260620 21:55:19.421046 18793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:39447
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:19.427469 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.419982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:44727 (local address 127.17.67.254:39009)
0620 21:55:19.420105 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.420109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.420123 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:19.420246 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.420249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.420295 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.420373 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.420979 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.421735 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.423064 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.423278 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.423355 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.424531 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.424533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.424535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.424555 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.425961 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.426335 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.426338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.426339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.426406 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.426688 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.426693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.426695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.426852 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.426954 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.427213 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.427328 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:55:19.427476 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.419727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.193:44727)
0620 21:55:19.420026 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.420043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.420128 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.420408 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.420414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.420665 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.420825 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.420835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.421871 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.421877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.422954 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.422967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.423392 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.423397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.423492 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.423950 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.423969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.424396 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.426062 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.426064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.426065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.426242 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.426503 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.426506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.426509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.426596 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.426957 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.426962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.427051 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.427153 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.427287 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:19.442553 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39009
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:19.552764 19006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:19.553057 19006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:19.553131 19006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:19.557381 19006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:19.557478 19006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:19.557554 19006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:19.557667 19006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:19.562999 19006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:19.564051 19006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:19.565081 19006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:19.574532 19012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.574532 19014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.574559 19011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.574798 19006 server_base.cc:1034] running on GCE node
I20260620 21:55:19.575487 19006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:19.576399 19006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:19.577646 19006 hybrid_clock.cc:648] HybridClock initialized: now 1781992519577617 us; error 49 us; skew 500 ppm
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:19.580825 19006 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:19.581858 19006 webserver.cc:460] Webserver started at http://127.17.67.194:46285/ using document root <none> and password file <none>
I20260620 21:55:19.582484 19006 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:19.582588 19006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:19.582832 19006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:19.585002 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "d876eb45e1d148ada111115cda13e13e"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "af99d717a6807f81d86224c5983e06e6"
server_key_iv: "c2524cb18300b569f413827a83ec346d"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.585752 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "d876eb45e1d148ada111115cda13e13e"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "af99d717a6807f81d86224c5983e06e6"
server_key_iv: "c2524cb18300b569f413827a83ec346d"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.589828 19006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:19.592271 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.593444 19006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:19.593621 19006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "d876eb45e1d148ada111115cda13e13e"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "af99d717a6807f81d86224c5983e06e6"
server_key_iv: "c2524cb18300b569f413827a83ec346d"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.593741 19006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:19.622969 19006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:19.625914 19006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:19.626150 19006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:19.627547 19006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:19.627620 19006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.627691 19006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:19.627725 19006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:19.638391 19006 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:36915
I20260620 21:55:19.638417 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:36915 every 8 connection(s)
I20260620 21:55:19.639407 19006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:19.641496 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.628481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:56023 (local address 127.17.67.254:39009)
0620 21:55:19.628632 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.628636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.629505 (+ 869us) server_negotiation.cc:407] Connection header received
0620 21:55:19.630627 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.630632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.630695 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.630805 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.632365 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.633254 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.634654 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.635060 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.635173 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.637892 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.637899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.637904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.637933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.639600 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.640199 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.640202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.640204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.640292 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.640614 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.640616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.640617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.640798 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.640873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.641289 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.641380 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:55:19.642294 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.628887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.194:56023)
0620 21:55:19.629335 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.629384 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.630370 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.631033 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.631043 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.631498 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.632180 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.632197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.633408 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.633411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.634521 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.634529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.635241 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.635244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.635385 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.635976 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.635991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.637722 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.639703 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.639708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.639718 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.640079 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.640391 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.640394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.640395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.640522 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.640884 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.640889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.641091 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.641274 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.641504 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260620 21:55:19.643639 19137 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39009
I20260620 21:55:19.643785 19137 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:19.644276 19137 heartbeater.cc:511] Master 127.17.67.254:39009 requested a full tablet report, sending...
I20260620 21:55:19.645301 18794 ts_manager.cc:194] Registered new tserver with Master: d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915)
I20260620 21:55:19.646129 18794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:56023
I20260620 21:55:19.649164 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19006
I20260620 21:55:19.649299 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:19.649618 17679 external_mini_cluster.cc:1428] Setting key 85b3fd3d8caa55abf2480eefb2142ccc
I20260620 21:55:19.653826 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.645801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:33761 (local address 127.17.67.254:39009)
0620 21:55:19.645894 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.645898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.646184 (+ 286us) server_negotiation.cc:407] Connection header received
0620 21:55:19.646312 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.646314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.646375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.646465 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.647222 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.647717 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.648596 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.648814 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.648943 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.650062 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.650065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.650067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.650087 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.651897 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.652357 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.652363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.652367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.652457 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.652756 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.652762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.652765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.653050 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.653172 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.653464 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.653614 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:55:19.653827 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.645665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.194:33761)
0620 21:55:19.646085 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.646099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.646208 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.646711 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.646714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.646918 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.647095 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.647103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.647867 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.647870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.648492 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.648500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.648931 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.648934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.649006 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.649508 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.649520 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.649917 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.652007 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.652010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.652012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.652254 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.652547 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.652549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.652550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.652619 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.653165 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.653169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.653274 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.653401 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.653592 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":9,"thread_start_us":225,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:19.674914 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39009
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:19.785630 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:19.785893 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:19.785938 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:19.789919 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:19.789978 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:19.790017 19146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:19.790051 19146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:19.795318 19146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:19.796494 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:19.797621 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:19.807642 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.807642 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:19.807868 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:19.807936 19146 server_base.cc:1034] running on GCE node
I20260620 21:55:19.808449 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:19.809082 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:19.810266 19146 hybrid_clock.cc:648] HybridClock initialized: now 1781992519810241 us; error 59 us; skew 500 ppm
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:19.813151 19146 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:19.814132 19146 webserver.cc:460] Webserver started at http://127.17.67.195:46633/ using document root <none> and password file <none>
I20260620 21:55:19.814698 19146 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:19.814788 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:19.815027 19146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:19.817018 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "3253aab225cc46b692723b59e03b0ad0"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "077f57cd969d954b1a3b29c0ece14262"
server_key_iv: "a411854acb1e68c3b6ab6ecafb9fdaf0"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.817706 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "3253aab225cc46b692723b59e03b0ad0"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "077f57cd969d954b1a3b29c0ece14262"
server_key_iv: "a411854acb1e68c3b6ab6ecafb9fdaf0"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.821666 19146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:19.824117 19161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.825469 19146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:19.825639 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "3253aab225cc46b692723b59e03b0ad0"
format_stamp: "Formatted at 2026-06-20 21:55:19 on dist-test-slave-cx8d"
server_key: "077f57cd969d954b1a3b29c0ece14262"
server_key_iv: "a411854acb1e68c3b6ab6ecafb9fdaf0"
server_key_version: "encryptionkey@0"
I20260620 21:55:19.825754 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:19.842603 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:19.845953 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:19.846159 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:19.847608 19146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:19.847689 19146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:19.847779 19146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:19.847827 19146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:19.859050 19146 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:35711
I20260620 21:55:19.859925 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:19.860678 19276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:35711 every 8 connection(s)
I20260620 21:55:19.861287 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19146
I20260620 21:55:19.861415 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:19.861732 17679 external_mini_cluster.cc:1428] Setting key 2d557de7bcb7bf61301103eac6cb6848
I20260620 21:55:19.861980 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.848424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:60933 (local address 127.17.67.254:39009)
0620 21:55:19.848562 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.848574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.850149 (+ 1575us) server_negotiation.cc:407] Connection header received
0620 21:55:19.851557 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.851561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.851635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.851725 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.853056 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.853771 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.854979 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.855275 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.855390 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.857993 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.857997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.858000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.858025 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.859581 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.860102 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.860106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.860108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.860195 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.860660 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.860665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.860666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.860870 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.860949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.861698 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.861831 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:19.862849 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.849486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.195:60933)
0620 21:55:19.849993 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.850035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.851292 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.851896 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.851905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.852444 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.852912 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.852925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.853892 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.853895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.854862 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.854871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.855434 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.855438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.855607 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.856188 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.856204 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.857828 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.859709 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.859715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.859730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.860014 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.860297 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.860300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.860302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.860496 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.860976 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.860981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.861289 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.861523 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.861900 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":128,"threads_started":1}
I20260620 21:55:19.864213 19277 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39009
I20260620 21:55:19.864367 19277 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:19.864919 19277 heartbeater.cc:511] Master 127.17.67.254:39009 requested a full tablet report, sending...
I20260620 21:55:19.865954 18794 ts_manager.cc:194] Registered new tserver with Master: 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711)
I20260620 21:55:19.866726 18794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:60933
I20260620 21:55:19.873781 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.866253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.17.67.195:40377)
0620 21:55:19.866523 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.866538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.866628 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.866898 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.866901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.867121 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:19.867258 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.867266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.867991 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.867995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.868574 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.868581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.869035 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.869038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.869128 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.869649 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:19.869663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:19.869975 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:19.872028 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.872034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.872036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.872281 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.872603 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:19.872607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:19.872609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:19.872717 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:19.873198 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:19.873202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:19.873298 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.873415 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.873587 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260620 21:55:19.874058 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.866314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:40377 (local address 127.17.67.254:39009)
0620 21:55:19.866441 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.866445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.866605 (+ 160us) server_negotiation.cc:407] Connection header received
0620 21:55:19.866716 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.866718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.866754 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.866863 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.867357 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.867870 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.868698 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.868925 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.869035 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.870115 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.870119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.870122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.870144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.871921 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.872391 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.872394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.872396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.872480 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.872862 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.872866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.872867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.873098 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.873207 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.873504 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.873942 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:55:19.875649 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:55:19 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:19.901999 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.893053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.17.67.254:39009)
0620 21:55:19.893195 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.893199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.893211 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:55:19.893253 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.893256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.893294 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.893370 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:19.894126 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.894634 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.895455 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.895660 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.895737 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.898533 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:19.898536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:19.898538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:19.898564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:19.900187 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.900660 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.900663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.900664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.900740 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:19.901071 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:19.901075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:19.901076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:19.901329 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:19.901445 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.901755 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.901862 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:55:19.905371 18794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:19.908633 18794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:19.925900 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.919467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40897 (local address 127.0.0.1:40982)
0620 21:55:19.919970 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.919983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.920056 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.921048 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.921049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.921064 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:19.921126 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.921129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922615 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.922618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.924031 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.924042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.925235 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.925239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.925368 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.925413 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.925536 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.925699 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 21:55:19.925901 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.919752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.17.67.193:40897)
0620 21:55:19.920225 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.920232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.920255 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:55:19.920339 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.920347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.920578 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.920753 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:19.921241 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922466 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.924197 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.925086 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.925206 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.925360 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.925643 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.925750 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":88,"threads_started":1}
I20260620 21:55:19.926509 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.919342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.17.67.194:36915)
0620 21:55:19.920017 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.920033 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.920060 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:55:19.920413 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.920420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.920622 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.920780 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:19.921295 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922460 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.924197 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.925086 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.925206 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.925375 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.926248 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.926353 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":121,"threads_started":1}
I20260620 21:55:19.926566 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.918511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:35711 (local address 127.0.0.1:60632)
0620 21:55:19.919314 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.919353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.919501 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.920758 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.920764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.920787 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:19.920947 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.920953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922240 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.922245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.924741 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.924749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.926146 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.926150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.926248 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.926271 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.926336 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.926416 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":1432,"thread_start_us":77,"threads_started":1}
I20260620 21:55:19.926956 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.918598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:36915 (local address 127.0.0.1:57824)
0620 21:55:19.919713 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.919726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.919808 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.920963 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.920965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.920981 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:19.921120 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.921129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922776 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.922780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.924031 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.924042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.925688 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.925692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.925819 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.926112 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.926704 (+ 592us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.926829 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":1,"thread_start_us":151,"threads_started":1}
I20260620 21:55:19.927089 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.918583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.17.67.195:35711)
0620 21:55:19.920319 (+ 1736us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.920331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.920350 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:55:19.920417 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.920421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.920577 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.920753 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:19.921074 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.922100 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.924880 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.925711 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.925834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.926781 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.926847 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.926950 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612,"thread_start_us":116,"threads_started":1}
I20260620 21:55:19.929508 19211 tablet_service.cc:1467] Processing CreateTablet for tablet e58314e7128643bf972251b49c053208 (DEFAULT_TABLE table=test-table [id=dd4914d3796f409099630bcf77ca5cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:19.929499 18931 tablet_service.cc:1467] Processing CreateTablet for tablet e58314e7128643bf972251b49c053208 (DEFAULT_TABLE table=test-table [id=dd4914d3796f409099630bcf77ca5cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:19.930737 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58314e7128643bf972251b49c053208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:19.930754 18931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58314e7128643bf972251b49c053208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:19.931717 19071 tablet_service.cc:1467] Processing CreateTablet for tablet e58314e7128643bf972251b49c053208 (DEFAULT_TABLE table=test-table [id=dd4914d3796f409099630bcf77ca5cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:19.932751 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58314e7128643bf972251b49c053208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:19.937552 19300 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Bootstrap starting.
I20260620 21:55:19.939308 19301 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Bootstrap starting.
I20260620 21:55:19.940011 19300 tablet_bootstrap.cc:654] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:19.940501 19302 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Bootstrap starting.
I20260620 21:55:19.941470 19300 log.cc:824] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:19.941766 19301 tablet_bootstrap.cc:654] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:19.942536 19301 log.cc:824] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:19.942827 19302 tablet_bootstrap.cc:654] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:19.943621 19300 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: No bootstrap required, opened a new log
I20260620 21:55:19.943692 19302 log.cc:824] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:19.943811 19300 ts_tablet_manager.cc:1397] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 21:55:19.944752 19301 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: No bootstrap required, opened a new log
I20260620 21:55:19.944941 19301 ts_tablet_manager.cc:1397] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 21:55:19.945544 19302 tablet_bootstrap.cc:492] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: No bootstrap required, opened a new log
I20260620 21:55:19.945832 19302 ts_tablet_manager.cc:1397] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 21:55:19.948174 19300 raft_consensus.cc:348] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.948460 19300 raft_consensus.cc:374] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:19.948510 19300 raft_consensus.cc:729] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9247270bac0f4af4adc2de99ddf82773, State: Initialized, Role: FOLLOWER
I20260620 21:55:19.948704 19301 raft_consensus.cc:348] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.948939 19301 raft_consensus.cc:374] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:19.948987 19301 raft_consensus.cc:729] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3253aab225cc46b692723b59e03b0ad0, State: Initialized, Role: FOLLOWER
I20260620 21:55:19.948999 19300 consensus_queue.cc:260] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.949318 19302 raft_consensus.cc:348] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.949433 19301 consensus_queue.cc:260] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.949569 19302 raft_consensus.cc:374] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:19.949649 19302 raft_consensus.cc:729] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d876eb45e1d148ada111115cda13e13e, State: Initialized, Role: FOLLOWER
I20260620 21:55:19.949828 18997 heartbeater.cc:503] Master 127.17.67.254:39009 was elected leader, sending a full tablet report...
I20260620 21:55:19.950003 19300 ts_tablet_manager.cc:1428] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260620 21:55:19.950112 19302 consensus_queue.cc:260] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.950335 19301 ts_tablet_manager.cc:1428] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 21:55:19.950645 19277 heartbeater.cc:503] Master 127.17.67.254:39009 was elected leader, sending a full tablet report...
I20260620 21:55:19.951135 19302 ts_tablet_manager.cc:1428] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 21:55:19.951563 19137 heartbeater.cc:503] Master 127.17.67.254:39009 was elected leader, sending a full tablet report...
I20260620 21:55:19.956055 19306 raft_consensus.cc:482] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:19.956238 19306 raft_consensus.cc:504] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.957840 19306 leader_election.cc:288] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711), d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915)
I20260620 21:55:19.963562 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.958057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:44947 (local address 127.17.67.195:35711)
0620 21:55:19.958205 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.958210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.958378 (+ 168us) server_negotiation.cc:407] Connection header received
0620 21:55:19.958524 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.958527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.958572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.958661 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:19.958940 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.959759 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.962246 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.962857 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.962985 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.963035 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.963363 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.963439 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:19.963562 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.958454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:42165 (local address 127.17.67.194:36915)
0620 21:55:19.958576 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.958580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.958593 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:19.958637 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.958643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.958685 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.958753 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:19.959029 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.959759 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.962246 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.962857 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.962985 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.963035 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.963378 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.963440 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:55:19.963616 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.958070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:36915 (local address 127.17.67.193:42165)
0620 21:55:19.958375 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.958388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.958464 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.958777 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.958780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.958797 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:19.958914 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.958923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.959886 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.959889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.962101 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.962113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.963022 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.963027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.963243 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.963280 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.963356 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.963406 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":172,"thread_start_us":93,"threads_started":1}
I20260620 21:55:19.963726 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.957997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:35711 (local address 127.17.67.193:44947)
0620 21:55:19.958276 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:19.958293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:19.958416 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:19.958668 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:19.958670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:19.958683 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:19.958838 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.958845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.959886 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.959889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.962101 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:19.962113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.963022 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.963027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:19.963164 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.963197 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 21:55:19.963301 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:55:19.963399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":186,"thread_start_us":121,"threads_started":1}
I20260620 21:55:19.964365 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58314e7128643bf972251b49c053208" candidate_uuid: "9247270bac0f4af4adc2de99ddf82773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3253aab225cc46b692723b59e03b0ad0" is_pre_election: true
I20260620 21:55:19.964363 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58314e7128643bf972251b49c053208" candidate_uuid: "9247270bac0f4af4adc2de99ddf82773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d876eb45e1d148ada111115cda13e13e" is_pre_election: true
I20260620 21:55:19.964648 19231 raft_consensus.cc:2454] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9247270bac0f4af4adc2de99ddf82773 in term 0.
I20260620 21:55:19.964709 19091 raft_consensus.cc:2454] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9247270bac0f4af4adc2de99ddf82773 in term 0.
I20260620 21:55:19.965204 18883 leader_election.cc:302] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, 9247270bac0f4af4adc2de99ddf82773, d876eb45e1d148ada111115cda13e13e; no voters:
I20260620 21:55:19.965503 19306 raft_consensus.cc:2781] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:19.965623 19306 raft_consensus.cc:482] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:19.965682 19306 raft_consensus.cc:3037] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:19.966727 19306 raft_consensus.cc:504] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.967121 19306 leader_election.cc:288] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [CANDIDATE]: Term 1 election: Requested vote from peers 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711), d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915)
I20260620 21:55:19.967427 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58314e7128643bf972251b49c053208" candidate_uuid: "9247270bac0f4af4adc2de99ddf82773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3253aab225cc46b692723b59e03b0ad0"
I20260620 21:55:19.967456 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58314e7128643bf972251b49c053208" candidate_uuid: "9247270bac0f4af4adc2de99ddf82773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d876eb45e1d148ada111115cda13e13e"
I20260620 21:55:19.967576 19231 raft_consensus.cc:3037] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:19.967577 19091 raft_consensus.cc:3037] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:19.968631 19231 raft_consensus.cc:2454] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9247270bac0f4af4adc2de99ddf82773 in term 1.
I20260620 21:55:19.968657 19091 raft_consensus.cc:2454] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9247270bac0f4af4adc2de99ddf82773 in term 1.
I20260620 21:55:19.968945 18884 leader_election.cc:302] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, 9247270bac0f4af4adc2de99ddf82773; no voters:
I20260620 21:55:19.969126 19306 raft_consensus.cc:2781] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:19.969349 19306 raft_consensus.cc:686] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 1 LEADER]: Becoming Leader. State: Replica: 9247270bac0f4af4adc2de99ddf82773, State: Running, Role: LEADER
I20260620 21:55:19.969720 19306 consensus_queue.cc:237] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } }
I20260620 21:55:19.972856 18794 catalog_manager.cc:5495] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "9247270bac0f4af4adc2de99ddf82773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } health_report { overall_health: HEALTHY } } }
I20260620 21:55:19.996502 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:19.992257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.17.67.193:40897)
0620 21:55:19.992417 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:19.992423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:19.992532 (+ 109us) server_negotiation.cc:407] Connection header received
0620 21:55:19.992831 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:19.992834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:19.992893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:19.993000 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:19.993275 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.993854 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.994798 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:19.995018 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:19.995133 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:19.995485 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:19.995908 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:19.996061 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:19.996202 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:55:19.996310 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:55:20.002712 19231 raft_consensus.cc:1261] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Refusing update from remote peer 9247270bac0f4af4adc2de99ddf82773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.002923 19091 raft_consensus.cc:1261] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Refusing update from remote peer 9247270bac0f4af4adc2de99ddf82773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.003608 19311 consensus_queue.cc:1035] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.003813 19306 consensus_queue.cc:1035] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.011572 19316 mvcc.cc:205] Tried to move back new op lower bound from 7299041361923616768 to 7299041361798524928. Current Snapshot: MvccSnapshot[applied={T|T < 7299041361923616768}]
I20260620 21:55:20.024878 18793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:20.025043 18793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:20.027601 18793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Sending DeleteTablet for 3 replicas of tablet e58314e7128643bf972251b49c053208
I20260620 21:55:20.028441 19071 tablet_service.cc:1514] Processing DeleteTablet for tablet e58314e7128643bf972251b49c053208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:57824
I20260620 21:55:20.028486 19211 tablet_service.cc:1514] Processing DeleteTablet for tablet e58314e7128643bf972251b49c053208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60632
I20260620 21:55:20.029021 18931 tablet_service.cc:1514] Processing DeleteTablet for tablet e58314e7128643bf972251b49c053208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:40982
I20260620 21:55:20.029088 19324 tablet_replica.cc:330] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: stopping tablet replica
I20260620 21:55:20.029429 19324 raft_consensus.cc:2229] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:20.029456 18793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:55:20.029796 19324 raft_consensus.cc:2258] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.029940 19326 tablet_replica.cc:330] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: stopping tablet replica
W20260620 21:55:20.029963 18793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:20.030320 19326 raft_consensus.cc:2229] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:20.030797 19326 raft_consensus.cc:2258] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.031288 19325 tablet_replica.cc:330] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: stopping tablet replica
I20260620 21:55:20.031579 19325 raft_consensus.cc:2229] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:20.031868 19325 raft_consensus.cc:2258] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.031881 19324 ts_tablet_manager.cc:1905] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.034155 19325 ts_tablet_manager.cc:1905] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.034993 19324 ts_tablet_manager.cc:1918] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:20.035084 19324 log.cc:1196] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/e58314e7128643bf972251b49c053208
I20260620 21:55:20.035406 19324 ts_tablet_manager.cc:1939] T e58314e7128643bf972251b49c053208 P 3253aab225cc46b692723b59e03b0ad0: Deleting consensus metadata
I20260620 21:55:20.035632 18931 tablet_service.cc:1467] Processing CreateTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (DEFAULT_TABLE table=test-table [id=7ac835d09a1b44148eeb876f5df71a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:20.035893 18931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.036572 18780 catalog_manager.cc:4841] TS 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711): tablet e58314e7128643bf972251b49c053208 (table test-table [id=dd4914d3796f409099630bcf77ca5cad]) successfully deleted
I20260620 21:55:20.036996 19325 ts_tablet_manager.cc:1918] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:20.036990 19211 tablet_service.cc:1467] Processing CreateTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (DEFAULT_TABLE table=test-table [id=7ac835d09a1b44148eeb876f5df71a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:20.037077 19325 log.cc:1196] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/e58314e7128643bf972251b49c053208
I20260620 21:55:20.037204 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.037387 19325 ts_tablet_manager.cc:1939] T e58314e7128643bf972251b49c053208 P d876eb45e1d148ada111115cda13e13e: Deleting consensus metadata
I20260620 21:55:20.038429 19300 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Bootstrap starting.
I20260620 21:55:20.039216 19300 tablet_bootstrap.cc:654] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.039285 19301 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Bootstrap starting.
I20260620 21:55:20.040123 19071 tablet_service.cc:1467] Processing CreateTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (DEFAULT_TABLE table=test-table [id=7ac835d09a1b44148eeb876f5df71a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:20.040652 18779 catalog_manager.cc:4841] TS d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915): tablet e58314e7128643bf972251b49c053208 (table test-table [id=dd4914d3796f409099630bcf77ca5cad]) successfully deleted
I20260620 21:55:20.041055 19300 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: No bootstrap required, opened a new log
I20260620 21:55:20.041150 19300 ts_tablet_manager.cc:1397] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:20.041481 19301 tablet_bootstrap.cc:654] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.041695 19300 raft_consensus.cc:348] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.041818 19300 raft_consensus.cc:374] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.041854 19300 raft_consensus.cc:729] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9247270bac0f4af4adc2de99ddf82773, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.041980 19300 consensus_queue.cc:260] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.042274 19300 ts_tablet_manager.cc:1428] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:20.040752 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.042974 19301 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: No bootstrap required, opened a new log
I20260620 21:55:20.043051 19301 ts_tablet_manager.cc:1397] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 21:55:20.043503 19301 raft_consensus.cc:348] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.043606 19301 raft_consensus.cc:374] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.043649 19301 raft_consensus.cc:729] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3253aab225cc46b692723b59e03b0ad0, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.043766 19301 consensus_queue.cc:260] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.044042 19301 ts_tablet_manager.cc:1428] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:20.045095 19302 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Bootstrap starting.
I20260620 21:55:20.046018 19302 tablet_bootstrap.cc:654] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.047076 19326 ts_tablet_manager.cc:1905] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.047363 19302 tablet_bootstrap.cc:492] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: No bootstrap required, opened a new log
I20260620 21:55:20.047449 19302 ts_tablet_manager.cc:1397] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:20.047921 19302 raft_consensus.cc:348] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.048025 19302 raft_consensus.cc:374] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.048058 19302 raft_consensus.cc:729] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d876eb45e1d148ada111115cda13e13e, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.048170 19302 consensus_queue.cc:260] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.048417 19302 ts_tablet_manager.cc:1428] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:20.049649 19326 ts_tablet_manager.cc:1918] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:20.049750 19326 log.cc:1196] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/e58314e7128643bf972251b49c053208
I20260620 21:55:20.050091 19326 ts_tablet_manager.cc:1939] T e58314e7128643bf972251b49c053208 P 9247270bac0f4af4adc2de99ddf82773: Deleting consensus metadata
I20260620 21:55:20.051286 18781 catalog_manager.cc:4841] TS 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897): tablet e58314e7128643bf972251b49c053208 (table test-table [id=dd4914d3796f409099630bcf77ca5cad]) successfully deleted
I20260620 21:55:20.087903 19308 raft_consensus.cc:482] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:20.088076 19308 raft_consensus.cc:504] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.089236 19308 leader_election.cc:288] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897), 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711)
I20260620 21:55:20.093871 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.089653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:38243 (local address 127.17.67.193:40897)
0620 21:55:20.089818 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.089823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.089988 (+ 165us) server_negotiation.cc:407] Connection header received
0620 21:55:20.090149 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.090152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.090202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.090291 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:20.090586 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.091362 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.092808 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.093220 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.093351 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.093407 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.093649 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.093738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:55:20.094089 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.089514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:35711 (local address 127.17.67.194:57915)
0620 21:55:20.089927 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.089940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.090035 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.090448 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.090450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.090463 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:20.090548 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.090555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.091309 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.091315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.092647 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.092657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.093669 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.093673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.093780 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.093801 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.093872 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.093943 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"spinlock_wait_cycles":10496,"thread_start_us":66,"threads_started":1}
I20260620 21:55:20.094267 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.089423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40897 (local address 127.17.67.194:38243)
0620 21:55:20.089887 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.089901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.090035 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.090320 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.090323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.090340 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:20.090475 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.090482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.091475 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.091478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.092647 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.092657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.093361 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.093364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.093484 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.093506 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.094057 (+ 551us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.094135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":3,"spinlock_wait_cycles":20736}
I20260620 21:55:20.094540 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.089713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:57915 (local address 127.17.67.195:35711)
0620 21:55:20.089846 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.089851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.090125 (+ 274us) server_negotiation.cc:407] Connection header received
0620 21:55:20.090185 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.090188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.090231 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.090328 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:20.090656 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.091196 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.092771 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.093402 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.093543 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.093612 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.094346 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.094426 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:55:20.094890 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d723bf43d543a2b7b81ba0d3a9e5f7" candidate_uuid: "d876eb45e1d148ada111115cda13e13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3253aab225cc46b692723b59e03b0ad0" is_pre_election: true
I20260620 21:55:20.094873 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d723bf43d543a2b7b81ba0d3a9e5f7" candidate_uuid: "d876eb45e1d148ada111115cda13e13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9247270bac0f4af4adc2de99ddf82773" is_pre_election: true
I20260620 21:55:20.095036 19231 raft_consensus.cc:2454] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d876eb45e1d148ada111115cda13e13e in term 0.
I20260620 21:55:20.095112 18951 raft_consensus.cc:2454] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d876eb45e1d148ada111115cda13e13e in term 0.
I20260620 21:55:20.095417 19024 leader_election.cc:302] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, d876eb45e1d148ada111115cda13e13e; no voters:
I20260620 21:55:20.095683 19308 raft_consensus.cc:2781] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:20.095749 19308 raft_consensus.cc:482] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:20.095788 19308 raft_consensus.cc:3037] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.097055 19308 raft_consensus.cc:504] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.097424 19308 leader_election.cc:288] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [CANDIDATE]: Term 1 election: Requested vote from peers 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897), 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711)
I20260620 21:55:20.097784 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d723bf43d543a2b7b81ba0d3a9e5f7" candidate_uuid: "d876eb45e1d148ada111115cda13e13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3253aab225cc46b692723b59e03b0ad0"
I20260620 21:55:20.097803 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d723bf43d543a2b7b81ba0d3a9e5f7" candidate_uuid: "d876eb45e1d148ada111115cda13e13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9247270bac0f4af4adc2de99ddf82773"
I20260620 21:55:20.097899 19231 raft_consensus.cc:3037] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.097967 18951 raft_consensus.cc:3037] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.098940 19231 raft_consensus.cc:2454] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d876eb45e1d148ada111115cda13e13e in term 1.
I20260620 21:55:20.099258 19024 leader_election.cc:302] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, d876eb45e1d148ada111115cda13e13e; no voters:
I20260620 21:55:20.099453 19308 raft_consensus.cc:2781] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:20.099568 18951 raft_consensus.cc:2454] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d876eb45e1d148ada111115cda13e13e in term 1.
I20260620 21:55:20.099606 19308 raft_consensus.cc:686] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 1 LEADER]: Becoming Leader. State: Replica: d876eb45e1d148ada111115cda13e13e, State: Running, Role: LEADER
I20260620 21:55:20.099844 19308 consensus_queue.cc:237] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.102072 18794 catalog_manager.cc:5495] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e reported cstate change: term changed from 0 to 1, leader changed from <none> to d876eb45e1d148ada111115cda13e13e (127.17.67.194). New cstate: current_term: 1 leader_uuid: "d876eb45e1d148ada111115cda13e13e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } health_report { overall_health: UNKNOWN } } }
W20260620 21:55:20.110908 19278 tablet.cc:2369] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:55:20 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:20.137368 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.127355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.17.67.254:39009)
0620 21:55:20.127507 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.127513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.127657 (+ 144us) server_negotiation.cc:407] Connection header received
0620 21:55:20.127778 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.127781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.127834 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.127937 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:20.128533 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.129317 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.130538 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.130885 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.130997 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.133787 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:20.133791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:20.133794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:20.133818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:20.135499 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.136038 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.136042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.136044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.136123 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.136420 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.136423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.136425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.136623 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:20.136716 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.137047 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.137212 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":55}
I20260620 21:55:20.137387 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.127239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39009 (local address 127.0.0.1:57798)
0620 21:55:20.127568 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.127581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.127679 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.127964 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.127966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.128319 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:20.128410 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.128415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.129424 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.129427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.130419 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.130429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.131008 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.131011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.131132 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.131793 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:20.131807 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:20.133651 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:20.135615 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:20.135625 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:20.135632 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:20.135944 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:20.136221 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:20.136224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:20.136226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:20.136331 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:20.136728 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:20.136735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:20.136864 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.136986 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.137148 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
W20260620 21:55:20.140691 19138 tablet.cc:2369] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:20.143267 18794 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:57798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:55:20.144021 18794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:20.149183 18931 tablet_service.cc:1467] Processing CreateTablet for tablet d718b2cb7e1f44a2bd18c09aa6675ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91d4655ed9b64fb0ad856a952b2d1334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:20.149281 19071 tablet_service.cc:1467] Processing CreateTablet for tablet d718b2cb7e1f44a2bd18c09aa6675ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91d4655ed9b64fb0ad856a952b2d1334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:20.149622 18931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718b2cb7e1f44a2bd18c09aa6675ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.149696 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718b2cb7e1f44a2bd18c09aa6675ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.152137 19300 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: Bootstrap starting.
I20260620 21:55:20.152762 19302 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: Bootstrap starting.
I20260620 21:55:20.153100 19300 tablet_bootstrap.cc:654] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.153359 19211 tablet_service.cc:1467] Processing CreateTablet for tablet d718b2cb7e1f44a2bd18c09aa6675ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91d4655ed9b64fb0ad856a952b2d1334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:20.153751 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718b2cb7e1f44a2bd18c09aa6675ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:20.153789 19302 tablet_bootstrap.cc:654] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.155241 19302 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: No bootstrap required, opened a new log
I20260620 21:55:20.155345 19302 ts_tablet_manager.cc:1397] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:20.155735 19302 raft_consensus.cc:348] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.155861 19302 raft_consensus.cc:374] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.155900 19302 raft_consensus.cc:729] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d876eb45e1d148ada111115cda13e13e, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.156045 19302 consensus_queue.cc:260] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.156298 19302 ts_tablet_manager.cc:1428] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:20.156941 19301 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: Bootstrap starting.
I20260620 21:55:20.157878 19301 tablet_bootstrap.cc:654] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:20.158614 19308 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.158774 19308 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.159188 19301 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: No bootstrap required, opened a new log
I20260620 21:55:20.159291 19301 ts_tablet_manager.cc:1397] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:20.159673 19301 raft_consensus.cc:348] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.159792 19301 raft_consensus.cc:374] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.159832 19301 raft_consensus.cc:729] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3253aab225cc46b692723b59e03b0ad0, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.159951 19301 consensus_queue.cc:260] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.160218 19301 ts_tablet_manager.cc:1428] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:20.160444 19307 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.160586 19307 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.161743 19300 tablet_bootstrap.cc:492] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: No bootstrap required, opened a new log
I20260620 21:55:20.161836 19300 ts_tablet_manager.cc:1397] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260620 21:55:20.162333 19300 raft_consensus.cc:348] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.162500 19300 raft_consensus.cc:374] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:20.162570 19300 raft_consensus.cc:729] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9247270bac0f4af4adc2de99ddf82773, State: Initialized, Role: FOLLOWER
I20260620 21:55:20.162756 19300 consensus_queue.cc:260] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.163043 19300 ts_tablet_manager.cc:1428] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:20.163975 19321 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.164115 19321 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 21:55:20.164307 18998 tablet.cc:2369] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:20.411888 19307 raft_consensus.cc:482] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:20.412076 19307 raft_consensus.cc:504] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.413281 19307 leader_election.cc:288] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897), d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915)
I20260620 21:55:20.418211 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.413722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:53009 (local address 127.17.67.193:40897)
0620 21:55:20.413944 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.413951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.414024 (+ 73us) server_negotiation.cc:407] Connection header received
0620 21:55:20.414265 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.414276 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.414328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.414415 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:20.414743 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.415537 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.416917 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.417631 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.417774 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.417855 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.418005 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.418092 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 21:55:20.418428 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.413966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48505 (local address 127.17.67.194:36915)
0620 21:55:20.414085 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.414090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.414178 (+ 88us) server_negotiation.cc:407] Connection header received
0620 21:55:20.414352 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.414356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.414451 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.414536 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:20.414917 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.415541 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.417098 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.417766 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.417880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.417953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.418199 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.418282 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:20.418689 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.413537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40897 (local address 127.17.67.195:53009)
0620 21:55:20.413934 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.413952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.414112 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.414452 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.414456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.414479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:20.414625 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.414634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.415666 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.415674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.416731 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.416741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.417752 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.417756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.417860 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.417886 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.418403 (+ 517us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.418479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":58,"thread_start_us":96,"threads_started":1}
I20260620 21:55:20.418825 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.413596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:36915 (local address 127.17.67.195:48505)
0620 21:55:20.414088 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.414101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.414197 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.414644 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.414647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.414667 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:20.414790 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.414796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.415690 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.415693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.416942 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.416954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.417935 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.417939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.418049 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.418080 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.418180 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.418257 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":42,"thread_start_us":77,"threads_started":1}
I20260620 21:55:20.419207 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718b2cb7e1f44a2bd18c09aa6675ae2" candidate_uuid: "3253aab225cc46b692723b59e03b0ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9247270bac0f4af4adc2de99ddf82773" is_pre_election: true
I20260620 21:55:20.419385 18951 raft_consensus.cc:2454] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3253aab225cc46b692723b59e03b0ad0 in term 0.
I20260620 21:55:20.419445 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718b2cb7e1f44a2bd18c09aa6675ae2" candidate_uuid: "3253aab225cc46b692723b59e03b0ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d876eb45e1d148ada111115cda13e13e" is_pre_election: true
I20260620 21:55:20.419606 19091 raft_consensus.cc:2454] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3253aab225cc46b692723b59e03b0ad0 in term 0.
I20260620 21:55:20.419842 19165 leader_election.cc:302] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, 9247270bac0f4af4adc2de99ddf82773; no voters:
I20260620 21:55:20.420136 19307 raft_consensus.cc:2781] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:20.420202 19307 raft_consensus.cc:482] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:20.420248 19307 raft_consensus.cc:3037] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.421396 19307 raft_consensus.cc:504] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.421838 19307 leader_election.cc:288] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [CANDIDATE]: Term 1 election: Requested vote from peers 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897), d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915)
I20260620 21:55:20.422142 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718b2cb7e1f44a2bd18c09aa6675ae2" candidate_uuid: "3253aab225cc46b692723b59e03b0ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9247270bac0f4af4adc2de99ddf82773"
I20260620 21:55:20.422159 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718b2cb7e1f44a2bd18c09aa6675ae2" candidate_uuid: "3253aab225cc46b692723b59e03b0ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d876eb45e1d148ada111115cda13e13e"
I20260620 21:55:20.422245 18951 raft_consensus.cc:3037] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.422258 19091 raft_consensus.cc:3037] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:20.423108 18951 raft_consensus.cc:2454] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3253aab225cc46b692723b59e03b0ad0 in term 1.
I20260620 21:55:20.423311 19091 raft_consensus.cc:2454] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3253aab225cc46b692723b59e03b0ad0 in term 1.
I20260620 21:55:20.423372 19165 leader_election.cc:302] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3253aab225cc46b692723b59e03b0ad0, 9247270bac0f4af4adc2de99ddf82773; no voters:
I20260620 21:55:20.423591 19307 raft_consensus.cc:2781] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:20.423820 19307 raft_consensus.cc:686] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 3253aab225cc46b692723b59e03b0ad0, State: Running, Role: LEADER
I20260620 21:55:20.424077 19307 consensus_queue.cc:237] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } }
I20260620 21:55:20.425333 19344 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: TxnStatusTablet state changed. Reason: New leader 3253aab225cc46b692723b59e03b0ad0. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.425463 19344 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:20.426643 18793 catalog_manager.cc:5495] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } health_report { overall_health: HEALTHY } } }
I20260620 21:55:20.429950 19346 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:55:20.430119 19346 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:55:20.452581 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.448682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:35711 (local address 127.0.0.1:60640)
0620 21:55:20.448876 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.448893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.449020 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.449354 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.449357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.449364 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:20.449474 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.449488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.450223 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.450227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.450916 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.450925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.451414 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.451418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.451537 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.451606 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.452285 (+ 679us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.452300 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.452371 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.452456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 21:55:20.452581 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.448776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.17.67.195:35711)
0620 21:55:20.448894 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.448899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.448945 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:55:20.449176 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.449178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.449240 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.449344 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:20.449612 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.450115 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.451034 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.451293 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.451413 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.451776 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.452187 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.452311 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.452381 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.452430 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:20.455704 19091 raft_consensus.cc:1261] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Refusing update from remote peer 3253aab225cc46b692723b59e03b0ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.455793 18951 raft_consensus.cc:1261] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Refusing update from remote peer 3253aab225cc46b692723b59e03b0ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.456256 19307 consensus_queue.cc:1035] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.456459 19344 consensus_queue.cc:1035] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.458025 19321 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: TxnStatusTablet state changed. Reason: New leader 3253aab225cc46b692723b59e03b0ad0. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.458142 19321 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.458076 19308 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: TxnStatusTablet state changed. Reason: New leader 3253aab225cc46b692723b59e03b0ad0. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.458184 19308 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.460850 19307 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.460980 19307 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:20.461261 19314 mvcc.cc:205] Tried to move back new op lower bound from 7299041363781038080 to 7299041363659661312. Current Snapshot: MvccSnapshot[applied={T|T < 7299041363781038080}]
I20260620 21:55:20.461412 19321 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.461498 19321 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 9247270bac0f4af4adc2de99ddf82773: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.463320 19315 mvcc.cc:205] Tried to move back new op lower bound from 7299041363781038080 to 7299041363659661312. Current Snapshot: MvccSnapshot[applied={T|T < 7299041363781038080}]
I20260620 21:55:20.463507 19308 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.463610 19308 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P d876eb45e1d148ada111115cda13e13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:20.460857 19344 tablet_replica.cc:439] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3253aab225cc46b692723b59e03b0ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 } } peers { permanent_uuid: "d876eb45e1d148ada111115cda13e13e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 36915 } } peers { permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 } } }
I20260620 21:55:20.464537 19344 tablet_replica.cc:442] T d718b2cb7e1f44a2bd18c09aa6675ae2 P 3253aab225cc46b692723b59e03b0ad0: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:20.470360 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.466283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.17.67.194:36915)
0620 21:55:20.466575 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.466579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.466588 (+ 9us) server_negotiation.cc:407] Connection header received
0620 21:55:20.466644 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.466646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.466692 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.466772 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:20.467045 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.467556 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.468703 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.468919 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.468999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.469402 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.469959 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.470059 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.470158 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.470208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260620 21:55:20.482447 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.478030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:35711 (local address 127.17.67.194:46841)
0620 21:55:20.478323 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.478337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.478423 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.478699 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.478702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.478709 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:20.478813 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.478818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.479857 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.479860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.480754 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.480764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.481420 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.481423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.481496 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.481542 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.482165 (+ 623us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.482172 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.482236 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.482306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
I20260620 21:55:20.482689 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.478121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:46841 (local address 127.17.67.195:35711)
0620 21:55:20.478253 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.478258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.478405 (+ 147us) server_negotiation.cc:407] Connection header received
0620 21:55:20.478515 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.478518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.478571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.478657 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:20.478925 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.479714 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.480903 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.481242 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.481352 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.481937 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.482022 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.482098 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.482508 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.482561 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:20.489972 19231 raft_consensus.cc:1261] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Refusing update from remote peer d876eb45e1d148ada111115cda13e13e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.489962 18951 raft_consensus.cc:1261] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Refusing update from remote peer d876eb45e1d148ada111115cda13e13e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:20.490504 19331 consensus_queue.cc:1035] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3253aab225cc46b692723b59e03b0ad0" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 35711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.490777 19308 consensus_queue.cc:1035] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9247270bac0f4af4adc2de99ddf82773" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:20.513909 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.510229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:54841 (local address 127.17.67.194:36915)
0620 21:55:20.510342 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.510346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.510592 (+ 246us) server_negotiation.cc:407] Connection header received
0620 21:55:20.510742 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.510745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.510790 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.510873 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:20.511253 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.511781 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.512645 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.512965 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.513067 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.513481 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.513548 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.513669 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.513762 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.513806 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 21:55:20.513990 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.510177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:36915 (local address 127.17.67.195:54841)
0620 21:55:20.510493 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:20.510511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:20.510643 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:20.510907 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:20.510910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:20.510917 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:20.511043 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.511048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.511888 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.511894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.512523 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:20.512531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.513172 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.513176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:20.513276 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.513338 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:20.513673 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:20.513679 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:20.513755 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:55:20.513835 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":113,"threads_started":1}
I20260620 21:55:20.535611 18793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:20.535785 18793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:20.537922 18793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fa8c21447a948e6a7171151dacb3727: Sending DeleteTablet for 3 replicas of tablet 83d723bf43d543a2b7b81ba0d3a9e5f7
I20260620 21:55:20.538425 18931 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:40982
I20260620 21:55:20.538602 19211 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60632
I20260620 21:55:20.538630 19326 tablet_replica.cc:330] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: stopping tablet replica
I20260620 21:55:20.538770 19326 raft_consensus.cc:2229] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:20.538908 19326 raft_consensus.cc:2258] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.538921 19362 tablet_replica.cc:330] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: stopping tablet replica
I20260620 21:55:20.539131 19362 raft_consensus.cc:2229] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:20.539155 19071 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:20 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:57824
I20260620 21:55:20.539328 19362 raft_consensus.cc:2258] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.539705 19326 ts_tablet_manager.cc:1905] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.540514 19363 tablet_replica.cc:330] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: stopping tablet replica
I20260620 21:55:20.540638 19363 raft_consensus.cc:2229] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:20.540928 19363 raft_consensus.cc:2258] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:20.541548 19363 ts_tablet_manager.cc:1905] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.542667 19326 ts_tablet_manager.cc:1918] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:20.542748 19326 log.cc:1196] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/83d723bf43d543a2b7b81ba0d3a9e5f7
I20260620 21:55:20.543020 19326 ts_tablet_manager.cc:1939] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 9247270bac0f4af4adc2de99ddf82773: Deleting consensus metadata
I20260620 21:55:20.543805 18781 catalog_manager.cc:4841] TS 9247270bac0f4af4adc2de99ddf82773 (127.17.67.193:40897): tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (table test-table [id=7ac835d09a1b44148eeb876f5df71a4d]) successfully deleted
I20260620 21:55:20.544308 19363 ts_tablet_manager.cc:1918] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:20.544380 19363 log.cc:1196] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/83d723bf43d543a2b7b81ba0d3a9e5f7
I20260620 21:55:20.544641 19363 ts_tablet_manager.cc:1939] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P d876eb45e1d148ada111115cda13e13e: Deleting consensus metadata
I20260620 21:55:20.544790 19362 ts_tablet_manager.cc:1905] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:20.545282 18779 catalog_manager.cc:4841] TS d876eb45e1d148ada111115cda13e13e (127.17.67.194:36915): tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (table test-table [id=7ac835d09a1b44148eeb876f5df71a4d]) successfully deleted
I20260620 21:55:20.547768 19362 ts_tablet_manager.cc:1918] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:20.547837 19362 log.cc:1196] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/83d723bf43d543a2b7b81ba0d3a9e5f7
I20260620 21:55:20.548094 19362 ts_tablet_manager.cc:1939] T 83d723bf43d543a2b7b81ba0d3a9e5f7 P 3253aab225cc46b692723b59e03b0ad0: Deleting consensus metadata
I20260620 21:55:20.548790 18780 catalog_manager.cc:4841] TS 3253aab225cc46b692723b59e03b0ad0 (127.17.67.195:35711): tablet 83d723bf43d543a2b7b81ba0d3a9e5f7 (table test-table [id=7ac835d09a1b44148eeb876f5df71a4d]) successfully deleted
Jun 20 21:55:20 dist-test-slave-cx8d krb5kdc[18749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:20.556491 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.540673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.17.67.193:40897)
0620 21:55:20.543768 (+ 3095us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.543773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.545886 (+ 2113us) server_negotiation.cc:407] Connection header received
0620 21:55:20.546092 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.546096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.546162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.546216 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:20.547229 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.547975 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.549119 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.549374 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.549433 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.552974 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:20.552985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:20.552996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:20.553021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:20.554741 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.555191 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.555196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.555198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.555275 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.555551 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.555556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.555558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.555804 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:20.556024 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.556203 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.556328 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3010}
W20260620 21:55:20.556938 18983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41004
I20260620 21:55:20.566435 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:20.558572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.17.67.254:39009)
0620 21:55:20.558707 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:20.558711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:20.559115 (+ 404us) server_negotiation.cc:407] Connection header received
0620 21:55:20.559279 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:20.559283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:20.559337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:20.559422 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:20.560050 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.560573 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.561358 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:20.561681 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:20.561791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:20.563137 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:20.563141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:20.563143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:20.563178 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:20.564677 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.565159 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.565163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.565165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.565251 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:20.565527 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:20.565532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:20.565534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:20.565774 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:20.565911 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:20.566149 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:55:20.566263 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 21:55:20.566946 18793 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812
I20260620 21:55:20.568243 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18866
I20260620 21:55:20.576536 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19006
I20260620 21:55:20.583675 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19146
I20260620 21:55:20.590183 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 18762
2026-06-20T21:55:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2796 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:20 dist-test-slave-cx8d krb5kdc[19370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:20 dist-test-slave-cx8d krb5kdc[19370](info): set up 2 sockets
Jun 20 21:55:20 dist-test-slave-cx8d krb5kdc[19370](info): commencing operation
krb5kdc: starting...
W20260620 21:55:21.705317 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:21 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:21.832544 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39773
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:21.947085 19383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:21.947366 19383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:21.947408 19383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:21.951597 19383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:21.951661 19383 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:55:21.951679 19383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:21.951697 19383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:21.951740 19383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:21.951771 19383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:21.957201 19383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:21.958334 19383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:21.959375 19383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:21.967950 19389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:21.967948 19388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:21.968039 19391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:21.968196 19383 server_base.cc:1034] running on GCE node
I20260620 21:55:21.968897 19383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:21.970086 19383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:21.971271 19383 hybrid_clock.cc:648] HybridClock initialized: now 1781992521971256 us; error 28 us; skew 500 ppm
Jun 20 21:55:21 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992521, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:21.974200 19383 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:21.975123 19383 webserver.cc:460] Webserver started at http://127.17.67.254:35489/ using document root <none> and password file <none>
I20260620 21:55:21.975646 19383 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:21.975705 19383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:21.975874 19383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:21.977896 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "7e9ea71c1e4045f6b08522768dc6c6c2"
format_stamp: "Formatted at 2026-06-20 21:55:21 on dist-test-slave-cx8d"
server_key: "5b75c1ae2d9c2b1102d718716008c37c"
server_key_iv: "c2a70b5992ba929a84fba3d93cc41611"
server_key_version: "encryptionkey@0"
I20260620 21:55:21.978499 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "7e9ea71c1e4045f6b08522768dc6c6c2"
format_stamp: "Formatted at 2026-06-20 21:55:21 on dist-test-slave-cx8d"
server_key: "5b75c1ae2d9c2b1102d718716008c37c"
server_key_iv: "c2a70b5992ba929a84fba3d93cc41611"
server_key_version: "encryptionkey@0"
I20260620 21:55:21.982401 19383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:55:21.984723 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:21.985978 19383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:55:21.986121 19383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "7e9ea71c1e4045f6b08522768dc6c6c2"
format_stamp: "Formatted at 2026-06-20 21:55:21 on dist-test-slave-cx8d"
server_key: "5b75c1ae2d9c2b1102d718716008c37c"
server_key_iv: "c2a70b5992ba929a84fba3d93cc41611"
server_key_version: "encryptionkey@0"
I20260620 21:55:21.986263 19383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:22.014825 19383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:22.018247 19383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:22.018462 19383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:22.026234 19383 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39773
I20260620 21:55:22.026247 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39773 every 8 connection(s)
I20260620 21:55:22.027148 19383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:22.028913 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19383
I20260620 21:55:22.029009 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:22.029304 17679 external_mini_cluster.cc:1428] Setting key 715feb8407b6013b28fd325b4a22e956
I20260620 21:55:22.030009 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:22.037181 19461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Bootstrap starting.
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:22.040282 19461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:22.041229 19461 log.cc:824] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:22.043731 19461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: No bootstrap required, opened a new log
I20260620 21:55:22.046188 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.030779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.17.67.254:39773)
0620 21:55:22.031228 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.031243 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.031280 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:22.031926 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.031943 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.032262 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.032531 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.033437 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.034910 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.036450 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.037317 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.037516 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.040790 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.040801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.040814 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.040843 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.043200 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.043647 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.043653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.043665 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.043775 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.044127 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.044130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.044132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.044519 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.044729 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.045021 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.045426 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260620 21:55:22.048283 19461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } }
I20260620 21:55:22.048542 19461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:22.048606 19461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9ea71c1e4045f6b08522768dc6c6c2, State: Initialized, Role: FOLLOWER
I20260620 21:55:22.049074 19461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } }
I20260620 21:55:22.049226 19461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:22.049314 19461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:22.049425 19461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:22.050509 19461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } }
I20260620 21:55:22.050879 19461 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9ea71c1e4045f6b08522768dc6c6c2; no voters:
I20260620 21:55:22.051234 19461 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:22.051295 19466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:22.051496 19466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9ea71c1e4045f6b08522768dc6c6c2, State: Running, Role: LEADER
I20260620 21:55:22.051787 19466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } }
I20260620 21:55:22.052484 19461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:22.053411 19467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9ea71c1e4045f6b08522768dc6c6c2. Latest consensus state: current_term: 1 leader_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } } }
I20260620 21:55:22.053546 19467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:22.053925 19468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9ea71c1e4045f6b08522768dc6c6c2" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39773 } } }
I20260620 21:55:22.053995 19475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:22.054016 19468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:22.057073 19475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:22.062748 19475 catalog_manager.cc:1347] Generated new cluster ID: b8c5495505e241ee9d84018efab1b78d
I20260620 21:55:22.062817 19475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:22.101377 19475 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:22.102393 19475 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:22.107816 19475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Generated new TSK 0
I20260620 21:55:22.108603 19475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:22.148590 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39773
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:22.266710 19487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:22.266973 19487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:22.267062 19487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:22.271353 19487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:22.271409 19487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:22.271448 19487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:55:22.271462 19487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:22.271494 19487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:22.276505 19487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:22.277418 19487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:22.278352 19487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:22.287855 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:22.287812 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:22.288111 19487 server_base.cc:1034] running on GCE node
W20260620 21:55:22.287839 19495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:22.288638 19487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:22.289472 19487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:22.290660 19487 hybrid_clock.cc:648] HybridClock initialized: now 1781992522290644 us; error 28 us; skew 500 ppm
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:22.293555 19487 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:22.294553 19487 webserver.cc:460] Webserver started at http://127.17.67.193:39763/ using document root <none> and password file <none>
I20260620 21:55:22.295112 19487 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:22.295172 19487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:22.295338 19487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:22.297425 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "ced139d319244b3e996bc35238b35071"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "78afaa6d6d460443709f06305263c727"
server_key_iv: "74d1553e30b8d15d0dd47fc42bb0fe35"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.298036 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "ced139d319244b3e996bc35238b35071"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "78afaa6d6d460443709f06305263c727"
server_key_iv: "74d1553e30b8d15d0dd47fc42bb0fe35"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.301946 19487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:22.304528 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.305900 19487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:22.305998 19487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "ced139d319244b3e996bc35238b35071"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "78afaa6d6d460443709f06305263c727"
server_key_iv: "74d1553e30b8d15d0dd47fc42bb0fe35"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.306097 19487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:22.320135 19487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:22.323491 19487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:22.323650 19487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:22.325062 19487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:22.325114 19487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.325205 19487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:22.325248 19487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:22.336581 19487 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:46805
I20260620 21:55:22.336681 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:46805 every 8 connection(s)
I20260620 21:55:22.337680 19487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:22.339951 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.326385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:38247 (local address 127.17.67.254:39773)
0620 21:55:22.326544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.326549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.327492 (+ 943us) server_negotiation.cc:407] Connection header received
0620 21:55:22.328944 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.328948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.329003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.329110 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.330833 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.331564 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.333000 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.333266 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.333381 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.336170 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.336173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.336175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.336201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.337820 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.338430 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.338433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.338435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.338527 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.338865 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.338870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.338871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.339054 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.339166 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.339656 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.339784 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:55:22.340710 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.326827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.193:38247)
0620 21:55:22.327327 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.327392 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.328705 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.329296 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.329306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.329846 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.330620 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.330645 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.331738 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.331742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.332871 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.332881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.333433 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.333437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.333688 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.334345 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.334367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.335976 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.337931 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.337937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.337951 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.338304 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.338627 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.338632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.338634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.338769 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.339163 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.339169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.339408 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.339616 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.339928 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260620 21:55:22.342068 19618 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39773
I20260620 21:55:22.342248 19618 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:22.342820 19618 heartbeater.cc:511] Master 127.17.67.254:39773 requested a full tablet report, sending...
I20260620 21:55:22.344408 19414 ts_manager.cc:194] Registered new tserver with Master: ced139d319244b3e996bc35238b35071 (127.17.67.193:46805)
I20260620 21:55:22.345639 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19487
I20260620 21:55:22.345754 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:22.346038 17679 external_mini_cluster.cc:1428] Setting key 52858047476c2e695ab52c1a7849ed0d
I20260620 21:55:22.346004 19414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:38247
I20260620 21:55:22.352496 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.344177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59237 (local address 127.17.67.254:39773)
0620 21:55:22.344294 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.344304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.344493 (+ 189us) server_negotiation.cc:407] Connection header received
0620 21:55:22.344644 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.344647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.344692 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.344762 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.345289 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.346031 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.347289 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.347650 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.347778 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.348973 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.348977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.348980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.349006 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.350587 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.351157 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.351160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.351162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.351240 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.351586 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.351589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.351590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.351782 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.351865 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.352201 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.352349 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:55:22.352625 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.344092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.193:59237)
0620 21:55:22.344406 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.344423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.344528 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.344806 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.344809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.345011 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.345132 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.345139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.346249 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.346252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.347158 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.347167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.347781 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.347785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.347893 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.348434 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.348446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.348826 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.350707 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.350712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.350715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.351013 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.351340 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.351346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.351351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.351458 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.351884 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.351890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.352015 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.352137 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.352314 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:22.376605 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39773
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:22.490825 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:22.491113 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:22.491165 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:22.495362 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:22.495422 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:22.495464 19627 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:55:22.495478 19627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:22.495510 19627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:22.500804 19627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:22.501879 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:22.503003 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:22.512670 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:22.512634 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:22.512658 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:22.513203 19627 server_base.cc:1034] running on GCE node
I20260620 21:55:22.513669 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:22.514343 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:22.515509 19627 hybrid_clock.cc:648] HybridClock initialized: now 1781992522515491 us; error 29 us; skew 500 ppm
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:22.518297 19627 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:22.519146 19627 webserver.cc:460] Webserver started at http://127.17.67.194:34611/ using document root <none> and password file <none>
I20260620 21:55:22.519735 19627 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:22.519825 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:22.520053 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:22.522141 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "be93c0f7a48e86527fe53b282d66b369"
server_key_iv: "1cc53662f0c8ab7e58b35d94a5223fdb"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.522768 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "be93c0f7a48e86527fe53b282d66b369"
server_key_iv: "1cc53662f0c8ab7e58b35d94a5223fdb"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.526819 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:22.529269 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.530630 19627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:22.530771 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "be93c0f7a48e86527fe53b282d66b369"
server_key_iv: "1cc53662f0c8ab7e58b35d94a5223fdb"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.530911 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:22.558374 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:22.562229 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:22.562455 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:22.564085 19627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:22.564178 19627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.564280 19627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:22.564335 19627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.575182 19627 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:43899
I20260620 21:55:22.575276 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:43899 every 8 connection(s)
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:22.576274 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:22.580173 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.565036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:52811 (local address 127.17.67.254:39773)
0620 21:55:22.565224 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.565229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.566208 (+ 979us) server_negotiation.cc:407] Connection header received
0620 21:55:22.567623 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.567628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.567690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.567751 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.569619 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.570620 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.572289 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.572700 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.572845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.576036 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.576041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.576044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.576085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.577890 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.578456 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.578459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.578461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.578547 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.578896 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.578899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.578901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.579129 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.579407 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.579911 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.580028 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 21:55:22.581043 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.565474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.194:52811)
0620 21:55:22.565981 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.566031 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.567354 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.568186 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.568192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.568684 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.569409 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.569429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.570762 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.570767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.572150 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.572160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.572914 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.572920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.573159 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.573921 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.573946 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.575807 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.578012 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.578017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.578029 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.578331 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.578643 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.578645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.578647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.578785 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.579425 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.579430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.579673 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.579827 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.580139 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":136,"threads_started":1}
I20260620 21:55:22.582424 19758 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39773
I20260620 21:55:22.582567 19758 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:22.583145 19758 heartbeater.cc:511] Master 127.17.67.254:39773 requested a full tablet report, sending...
I20260620 21:55:22.583674 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19627
I20260620 21:55:22.583808 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:22.584148 17679 external_mini_cluster.cc:1428] Setting key 94b9eadd8ea4ac7855cf1102074c9943
I20260620 21:55:22.584335 19415 ts_manager.cc:194] Registered new tserver with Master: 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899)
I20260620 21:55:22.585251 19415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:52811
I20260620 21:55:22.593544 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.584868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:60971 (local address 127.17.67.254:39773)
0620 21:55:22.584970 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.584974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.585763 (+ 789us) server_negotiation.cc:407] Connection header received
0620 21:55:22.585879 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.585882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.585938 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.586022 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.586667 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.587407 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.588776 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.589022 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.589191 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.590364 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.590368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.590371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.590393 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.591851 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.592282 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.592288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.592290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.592376 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.592696 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.592699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.592701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.592915 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.593022 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.593299 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.593407 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:55:22.593559 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.584820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.194:60971)
0620 21:55:22.585664 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.585680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.585787 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.586068 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.586073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.586342 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.586515 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.586527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.587523 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.587530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.588652 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.588666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.589151 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.589157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.589257 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.589881 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.589895 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.590227 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.591962 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.591967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.591969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.592180 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.592474 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.592477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.592479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.592572 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.593022 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.593025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.593122 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.593225 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.593372 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:22.610752 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39773
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:22.724437 19767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:22.724699 19767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:22.724750 19767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:22.729338 19767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:22.729399 19767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:22.729439 19767 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:55:22.729453 19767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:22.729485 19767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:22.734655 19767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:22.735648 19767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:22.736660 19767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:22.746424 19775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:22.746411 19773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:22.746411 19772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:22.746821 19767 server_base.cc:1034] running on GCE node
I20260620 21:55:22.747257 19767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:22.747942 19767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:22.749177 19767 hybrid_clock.cc:648] HybridClock initialized: now 1781992522749157 us; error 33 us; skew 500 ppm
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:22.752199 19767 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:22.753265 19767 webserver.cc:460] Webserver started at http://127.17.67.195:33901/ using document root <none> and password file <none>
I20260620 21:55:22.753901 19767 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:22.753993 19767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:22.754231 19767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:22.756316 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "6487b8e12442438c8336645a8abe1e60"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "86a18cd792867e1878bfdc6d394d46b7"
server_key_iv: "2b00d5277c0c93395d9301dafe54b12b"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.756877 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "6487b8e12442438c8336645a8abe1e60"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "86a18cd792867e1878bfdc6d394d46b7"
server_key_iv: "2b00d5277c0c93395d9301dafe54b12b"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.760589 19767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:55:22.763046 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.764273 19767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 21:55:22.764413 19767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "6487b8e12442438c8336645a8abe1e60"
format_stamp: "Formatted at 2026-06-20 21:55:22 on dist-test-slave-cx8d"
server_key: "86a18cd792867e1878bfdc6d394d46b7"
server_key_iv: "2b00d5277c0c93395d9301dafe54b12b"
server_key_version: "encryptionkey@0"
I20260620 21:55:22.764528 19767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:22.780414 19767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:22.783869 19767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:22.784032 19767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:22.785380 19767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:22.785458 19767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.785523 19767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:22.785597 19767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:22.796403 19767 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:37453
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:22.797192 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:37453 every 8 connection(s)
I20260620 21:55:22.797425 19767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:22.801833 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.786416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:49735 (local address 127.17.67.254:39773)
0620 21:55:22.786543 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.786563 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.787741 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 21:55:22.789078 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.789082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.789138 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.789233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.791235 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.792230 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.794006 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.794400 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.794539 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.797772 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.797779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.797784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.797820 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.799577 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.800155 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.800159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.800161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.800256 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.800627 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.800630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.800633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.800845 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.800962 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.801482 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.801640 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:22.802635 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.786996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.195:49735)
0620 21:55:22.787559 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.787603 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.788794 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.789606 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.789618 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.790285 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.791037 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.791070 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.792365 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.792371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.793855 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.793865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.794578 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.794583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.794842 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.795537 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.795557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.797546 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.799729 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.799736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.799752 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.800035 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.800364 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.800367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.800370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.800517 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.800958 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.800965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.801230 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.801426 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.801789 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":168,"threads_started":1}
I20260620 21:55:22.803880 19898 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39773
I20260620 21:55:22.804029 19898 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:22.804607 19898 heartbeater.cc:511] Master 127.17.67.254:39773 requested a full tablet report, sending...
I20260620 21:55:22.805701 19415 ts_manager.cc:194] Registered new tserver with Master: 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:22.806614 19415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:49735
I20260620 21:55:22.807152 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 19767
I20260620 21:55:22.807272 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:22.807549 17679 external_mini_cluster.cc:1428] Setting key ac8ba6fdb8ac54325295f64713676c9d
I20260620 21:55:22.809732 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:55:22.814383 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.806053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.17.67.195:34719)
0620 21:55:22.806388 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.806400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.806476 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.806757 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.806760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.806979 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:22.807103 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.807110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.808117 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.808122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.809046 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.809057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.809515 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.809519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.809642 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.810136 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:22.810150 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:22.810587 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:22.812802 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.812805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.812807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.812992 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.813303 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:22.813307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:22.813308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:22.813394 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:22.813844 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:22.813848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:22.813941 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.814048 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.814184 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260620 21:55:22.814391 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.806186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:34719 (local address 127.17.67.254:39773)
0620 21:55:22.806299 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.806304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.806466 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:55:22.806553 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.806556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.806601 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.806687 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.807251 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.807978 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.809157 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.809395 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.809512 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.810784 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.810791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.810798 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.810820 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.812654 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.813100 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.813106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.813111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.813202 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.813485 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.813489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.813491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.813739 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.813844 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.814107 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.814261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:55:22 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:22.837520 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.829103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.17.67.254:39773)
0620 21:55:22.829257 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.829261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.829272 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:22.829309 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.829312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.829348 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.829413 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:22.830228 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.830728 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.831611 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.831825 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.831906 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.834387 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:22.834394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:22.834400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:22.834428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:22.835885 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.836301 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.836306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.836308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.836384 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:22.836701 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:22.836705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:22.836707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:22.836872 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:22.837015 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.837243 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.837355 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:22.840391 19415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:22.843492 19415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:22.857854 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.852015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:37453 (local address 127.0.0.1:60302)
0620 21:55:22.852474 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.852486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.852555 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.853711 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.853717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.853741 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:22.853848 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.853854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.855055 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.855058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.856378 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.856386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857256 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857269 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.857382 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857422 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.857531 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.857694 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":83,"thread_start_us":148,"threads_started":1}
I20260620 21:55:22.857867 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.851793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46805 (local address 127.0.0.1:38478)
0620 21:55:22.852318 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.852366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.852496 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.853220 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.853226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.853249 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:22.853433 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.853442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.854975 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.854980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.856482 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.856492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857530 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.857645 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857667 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.857726 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.857780 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":142,"thread_start_us":66,"threads_started":1}
I20260620 21:55:22.858007 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.851957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.17.67.193:46805)
0620 21:55:22.852403 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.852413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.852441 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:55:22.852720 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.852726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.852951 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.853127 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:22.853682 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.854851 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.856627 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857420 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857536 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857700 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.857757 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.857847 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":92,"threads_started":1}
I20260620 21:55:22.858328 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.852503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.0.0.1:33110)
0620 21:55:22.852744 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.852756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.852979 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.853914 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.853917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.853936 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:22.854024 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.854030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.855391 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.855394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.856594 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.856601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857775 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.857863 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857884 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.857955 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.858205 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
I20260620 21:55:22.858635 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.852843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.17.67.195:37453)
0620 21:55:22.853233 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.853244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.853267 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:55:22.853341 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.853355 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.853528 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.853986 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:22.854114 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.854874 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.856516 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857132 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857260 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857426 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.858298 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.858503 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":88,"threads_started":1}
I20260620 21:55:22.859432 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.852885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.17.67.194:43899)
0620 21:55:22.853355 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.853367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.853387 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:22.853450 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.853454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.853647 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.853780 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:22.854194 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.855276 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.856713 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.857451 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.857540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.857714 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.858569 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.858641 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":101,"threads_started":1}
I20260620 21:55:22.861238 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7008b7d1604e259b1f01d781046293 (DEFAULT_TABLE table=test-table [id=a8cf3846941d401d8df4132bd3bf81b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:22.861231 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7008b7d1604e259b1f01d781046293 (DEFAULT_TABLE table=test-table [id=a8cf3846941d401d8df4132bd3bf81b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:22.861555 19681 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7008b7d1604e259b1f01d781046293 (DEFAULT_TABLE table=test-table [id=a8cf3846941d401d8df4132bd3bf81b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:22.862380 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7008b7d1604e259b1f01d781046293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:22.862375 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7008b7d1604e259b1f01d781046293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:22.862548 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f7008b7d1604e259b1f01d781046293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:22.869305 19921 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Bootstrap starting.
I20260620 21:55:22.870208 19923 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Bootstrap starting.
I20260620 21:55:22.871434 19921 tablet_bootstrap.cc:654] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:22.871865 19923 tablet_bootstrap.cc:654] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:22.872189 19921 log.cc:824] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:22.872596 19923 log.cc:824] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:22.873170 19922 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Bootstrap starting.
I20260620 21:55:22.874449 19921 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: No bootstrap required, opened a new log
I20260620 21:55:22.874648 19921 ts_tablet_manager.cc:1397] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:22.874720 19923 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: No bootstrap required, opened a new log
I20260620 21:55:22.874893 19923 ts_tablet_manager.cc:1397] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:55:22.875151 19922 tablet_bootstrap.cc:654] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:22.875887 19922 log.cc:824] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:22.877768 19922 tablet_bootstrap.cc:492] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: No bootstrap required, opened a new log
I20260620 21:55:22.877946 19922 ts_tablet_manager.cc:1397] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:55:22.877938 19921 raft_consensus.cc:348] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.878208 19921 raft_consensus.cc:374] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:22.878263 19921 raft_consensus.cc:729] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Initialized, Role: FOLLOWER
I20260620 21:55:22.878697 19921 consensus_queue.cc:260] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.878620 19923 raft_consensus.cc:348] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.878841 19923 raft_consensus.cc:374] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:22.878904 19923 raft_consensus.cc:729] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Initialized, Role: FOLLOWER
I20260620 21:55:22.879300 19923 consensus_queue.cc:260] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.879642 19921 ts_tablet_manager.cc:1428] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:55:22.879958 19898 heartbeater.cc:503] Master 127.17.67.254:39773 was elected leader, sending a full tablet report...
I20260620 21:55:22.880200 19923 ts_tablet_manager.cc:1428] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:55:22.881125 19758 heartbeater.cc:503] Master 127.17.67.254:39773 was elected leader, sending a full tablet report...
I20260620 21:55:22.881856 19922 raft_consensus.cc:348] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.882162 19922 raft_consensus.cc:374] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:22.882236 19922 raft_consensus.cc:729] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced139d319244b3e996bc35238b35071, State: Initialized, Role: FOLLOWER
I20260620 21:55:22.882673 19922 consensus_queue.cc:260] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.883327 19618 heartbeater.cc:503] Master 127.17.67.254:39773 was elected leader, sending a full tablet report...
I20260620 21:55:22.883697 19922 ts_tablet_manager.cc:1428] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:55:22.989967 19928 raft_consensus.cc:482] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:22.990236 19928 raft_consensus.cc:504] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.991530 19928 leader_election.cc:288] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899)
I20260620 21:55:22.995862 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.991869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46805 (local address 127.17.67.195:37837)
0620 21:55:22.992041 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.992061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.992209 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.992543 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.992546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.992566 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:22.992705 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.992715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.993621 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.993627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.994678 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.994687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.995476 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.995483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.995609 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.995629 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.995695 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.995749 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260620 21:55:22.995949 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.992006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:37837 (local address 127.17.67.193:46805)
0620 21:55:22.992147 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.992153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.992180 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:55:22.992345 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.992351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.992418 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.992512 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:22.992843 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.993485 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.994836 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.995338 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.995454 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.995561 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.995737 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.995816 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:55:22.996760 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7008b7d1604e259b1f01d781046293" candidate_uuid: "6487b8e12442438c8336645a8abe1e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071" is_pre_election: true
I20260620 21:55:22.996850 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.992011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:53371 (local address 127.17.67.194:43899)
0620 21:55:22.992142 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:22.992147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:22.992749 (+ 602us) server_negotiation.cc:407] Connection header received
0620 21:55:22.992868 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:22.992871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:22.992923 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:22.993009 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:22.993723 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.994356 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.995570 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.996246 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.996397 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.996449 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:22.996660 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 21:55:22.996736 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:22.996871 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:22.991869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.17.67.195:53371)
0620 21:55:22.992616 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:22.992630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:22.992760 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:22.993026 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:22.993028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:22.993053 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:22.993556 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.993568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.994486 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.994490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.995377 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:22.995385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:22.996397 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:22.996402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:22.996521 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:22.996539 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:55:22.996644 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:55:22.996713 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":303,"thread_start_us":183,"threads_started":1}
I20260620 21:55:22.997133 19571 raft_consensus.cc:2454] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6487b8e12442438c8336645a8abe1e60 in term 0.
I20260620 21:55:22.997542 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7008b7d1604e259b1f01d781046293" candidate_uuid: "6487b8e12442438c8336645a8abe1e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" is_pre_election: true
I20260620 21:55:22.997749 19784 leader_election.cc:302] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6487b8e12442438c8336645a8abe1e60, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:22.997852 19712 raft_consensus.cc:2454] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6487b8e12442438c8336645a8abe1e60 in term 0.
I20260620 21:55:22.998101 19928 raft_consensus.cc:2781] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:22.998212 19928 raft_consensus.cc:482] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:22.998294 19928 raft_consensus.cc:3037] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:22.999405 19928 raft_consensus.cc:504] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:22.999756 19928 leader_election.cc:288] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [CANDIDATE]: Term 1 election: Requested vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899)
I20260620 21:55:23.000082 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7008b7d1604e259b1f01d781046293" candidate_uuid: "6487b8e12442438c8336645a8abe1e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071"
I20260620 21:55:23.000113 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f7008b7d1604e259b1f01d781046293" candidate_uuid: "6487b8e12442438c8336645a8abe1e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
I20260620 21:55:23.000224 19571 raft_consensus.cc:3037] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.000243 19712 raft_consensus.cc:3037] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.001654 19712 raft_consensus.cc:2454] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6487b8e12442438c8336645a8abe1e60 in term 1.
I20260620 21:55:23.001652 19571 raft_consensus.cc:2454] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6487b8e12442438c8336645a8abe1e60 in term 1.
I20260620 21:55:23.001953 19783 leader_election.cc:302] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, 6487b8e12442438c8336645a8abe1e60; no voters:
I20260620 21:55:23.002159 19928 raft_consensus.cc:2781] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:23.002405 19928 raft_consensus.cc:686] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 1 LEADER]: Becoming Leader. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Running, Role: LEADER
I20260620 21:55:23.002760 19928 consensus_queue.cc:237] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } }
I20260620 21:55:23.006240 19412 catalog_manager.cc:5495] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6487b8e12442438c8336645a8abe1e60 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "6487b8e12442438c8336645a8abe1e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } health_report { overall_health: UNKNOWN } } }
W20260620 21:55:23.048421 19899 tablet.cc:2369] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:55:23 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992521, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
W20260620 21:55:23.077725 19759 tablet.cc:2369] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:23.078210 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.067980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39773 (local address 127.0.0.1:41646)
0620 21:55:23.068279 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.068297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.068439 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.068705 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.068708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.069188 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:23.069289 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.069296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.070295 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.070302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.070924 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.070932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.071447 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.071451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.071556 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.072191 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:23.072204 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:23.074145 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:23.076392 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:23.076408 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:23.076421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:23.076676 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:23.076992 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:23.076998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:23.077005 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:23.077083 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:23.077507 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:23.077516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:23.077737 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.077882 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.078065 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260620 21:55:23.078346 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.068044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.17.67.254:39773)
0620 21:55:23.068180 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.068184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.068375 (+ 191us) server_negotiation.cc:407] Connection header received
0620 21:55:23.068536 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.068540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.068592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.068684 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:23.069403 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.070143 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.071067 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.071319 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.071433 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.074398 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:23.074402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:23.074405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:23.074431 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:23.076233 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:23.076791 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:23.076796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:23.076798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:23.076885 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:23.077185 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:23.077190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:23.077192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:23.077388 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:23.077506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.077981 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.078164 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":46}
I20260620 21:55:23.083086 19412 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:41646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:55:23.084234 19412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 21:55:23.089044 19619 tablet.cc:2369] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:23.090058 19552 tablet_service.cc:1467] Processing CreateTablet for tablet ce5b804136e04444be8df3d66ba5c87e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10db81bf988046fda6f4ba9b47dfeed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:23.090253 19681 tablet_service.cc:1467] Processing CreateTablet for tablet ce5b804136e04444be8df3d66ba5c87e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10db81bf988046fda6f4ba9b47dfeed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:23.090310 19832 tablet_service.cc:1467] Processing CreateTablet for tablet ce5b804136e04444be8df3d66ba5c87e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10db81bf988046fda6f4ba9b47dfeed0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:23.090878 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5b804136e04444be8df3d66ba5c87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.090878 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5b804136e04444be8df3d66ba5c87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.091075 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce5b804136e04444be8df3d66ba5c87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.093755 19923 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: Bootstrap starting.
I20260620 21:55:23.094234 19922 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: Bootstrap starting.
I20260620 21:55:23.094269 19921 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: Bootstrap starting.
I20260620 21:55:23.094939 19923 tablet_bootstrap.cc:654] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.095052 19922 tablet_bootstrap.cc:654] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.095408 19921 tablet_bootstrap.cc:654] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.096446 19922 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: No bootstrap required, opened a new log
I20260620 21:55:23.096535 19923 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: No bootstrap required, opened a new log
I20260620 21:55:23.096555 19922 ts_tablet_manager.cc:1397] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:23.096632 19923 ts_tablet_manager.cc:1397] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:23.097045 19922 raft_consensus.cc:348] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.097201 19922 raft_consensus.cc:374] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.097245 19922 raft_consensus.cc:729] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced139d319244b3e996bc35238b35071, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.097139 19923 raft_consensus.cc:348] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.097291 19923 raft_consensus.cc:374] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.097342 19923 raft_consensus.cc:729] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.097376 19922 consensus_queue.cc:260] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.097494 19921 tablet_bootstrap.cc:492] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: No bootstrap required, opened a new log
I20260620 21:55:23.097599 19921 ts_tablet_manager.cc:1397] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:23.097688 19922 ts_tablet_manager.cc:1428] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 21:55:23.097496 19923 consensus_queue.cc:260] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.097955 19923 ts_tablet_manager.cc:1428] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:23.097898 19929 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.098045 19929 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.098019 19921 raft_consensus.cc:348] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.098138 19921 raft_consensus.cc:374] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.098184 19921 raft_consensus.cc:729] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.098331 19921 consensus_queue.cc:260] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.098606 19921 ts_tablet_manager.cc:1428] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:23.098574 19927 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.098707 19927 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.099184 19928 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.099308 19928 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.305486 19927 raft_consensus.cc:482] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:23.305660 19927 raft_consensus.cc:504] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.306936 19927 leader_election.cc:288] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:23.311143 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.307333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:59529 (local address 127.17.67.193:46805)
0620 21:55:23.307460 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.307465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.307595 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:55:23.307758 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.307761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.307812 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.307892 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:23.308149 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.308753 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310119 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.310590 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310713 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.310769 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.310945 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.311011 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:55:23.311170 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.307170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46805 (local address 127.17.67.194:59529)
0620 21:55:23.307497 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.307520 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.307653 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.307914 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.307916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.307931 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:23.308047 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.308051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.308876 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.308879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.310019 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.310031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.310709 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.310813 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.310835 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.310930 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.310995 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260620 21:55:23.311323 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.307317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58497 (local address 127.17.67.195:37453)
0620 21:55:23.307460 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.307465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.307645 (+ 180us) server_negotiation.cc:407] Connection header received
0620 21:55:23.307836 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.307847 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.307890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.307981 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:23.308305 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.308888 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310184 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.310700 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310851 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.310902 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.311119 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.311197 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:55:23.311342 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.307213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:37453 (local address 127.17.67.194:58497)
0620 21:55:23.307544 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.307557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.307715 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.308001 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.308005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.308021 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:23.308151 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.308156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.309015 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.309022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.310051 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.310061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.310853 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.310857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.310967 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.310986 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.311104 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.311184 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":78,"thread_start_us":66,"threads_started":1}
I20260620 21:55:23.311628 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5b804136e04444be8df3d66ba5c87e" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071" is_pre_election: true
I20260620 21:55:23.311800 19571 raft_consensus.cc:2454] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 0.
I20260620 21:55:23.311892 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5b804136e04444be8df3d66ba5c87e" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60" is_pre_election: true
I20260620 21:55:23.312144 19852 raft_consensus.cc:2454] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 0.
I20260620 21:55:23.312194 19644 leader_election.cc:302] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:23.312530 19927 raft_consensus.cc:2781] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:23.312606 19927 raft_consensus.cc:482] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:23.312661 19927 raft_consensus.cc:3037] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.313715 19927 raft_consensus.cc:504] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.314054 19927 leader_election.cc:288] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 election: Requested vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:23.314407 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5b804136e04444be8df3d66ba5c87e" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60"
I20260620 21:55:23.314407 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce5b804136e04444be8df3d66ba5c87e" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071"
I20260620 21:55:23.314531 19571 raft_consensus.cc:3037] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.314531 19852 raft_consensus.cc:3037] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.315642 19571 raft_consensus.cc:2454] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 1.
I20260620 21:55:23.315642 19852 raft_consensus.cc:2454] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 1.
I20260620 21:55:23.315898 19644 leader_election.cc:302] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:23.316072 19927 raft_consensus.cc:2781] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:23.316260 19927 raft_consensus.cc:686] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 LEADER]: Becoming Leader. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Running, Role: LEADER
I20260620 21:55:23.316604 19927 consensus_queue.cc:237] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.319054 19944 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: TxnStatusTablet state changed. Reason: New leader 4e9b12fd0bb045ccbe4403c77816fa4b. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.319192 19944 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:23.319447 19946 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:55:23.319649 19946 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:55:23.320410 19415 catalog_manager.cc:5495] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194). New cstate: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:23.338128 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.334015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.17.67.194:43899)
0620 21:55:23.334139 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.334144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.334160 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:55:23.334332 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.334335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.334401 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.334514 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.334843 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.335768 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.336671 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.336898 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.337005 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.337279 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.337770 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.337886 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.337952 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.338003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:23.338140 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.333934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.0.0.1:33118)
0620 21:55:23.334075 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.334093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.334192 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.334595 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.334601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.334610 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:23.334733 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.334738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.335872 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.335874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.336555 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.336564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.337005 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.337008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.337099 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.337139 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.337860 (+ 721us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.337875 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.337944 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.338027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 21:55:23.344244 19852 raft_consensus.cc:1261] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Refusing update from remote peer 4e9b12fd0bb045ccbe4403c77816fa4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.344388 19571 raft_consensus.cc:1261] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Refusing update from remote peer 4e9b12fd0bb045ccbe4403c77816fa4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.344905 19944 consensus_queue.cc:1035] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.345068 19927 consensus_queue.cc:1035] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.348500 19928 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: TxnStatusTablet state changed. Reason: New leader 4e9b12fd0bb045ccbe4403c77816fa4b. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.348626 19928 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.348798 19929 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: TxnStatusTablet state changed. Reason: New leader 4e9b12fd0bb045ccbe4403c77816fa4b. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.348904 19929 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.352411 19927 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.352530 19927 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:23.352598 19928 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.352717 19928 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 6487b8e12442438c8336645a8abe1e60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.354621 19949 mvcc.cc:205] Tried to move back new op lower bound from 7299041375611142144 to 7299041375507320832. Current Snapshot: MvccSnapshot[applied={T|T < 7299041375611142144}]
I20260620 21:55:23.356798 19944 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.356905 19944 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P 4e9b12fd0bb045ccbe4403c77816fa4b: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:23.358711 19929 tablet_replica.cc:439] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } } }
I20260620 21:55:23.358824 19929 tablet_replica.cc:442] T ce5b804136e04444be8df3d66ba5c87e P ced139d319244b3e996bc35238b35071: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:23.358944 19950 mvcc.cc:205] Tried to move back new op lower bound from 7299041375611142144 to 7299041375507320832. Current Snapshot: MvccSnapshot[applied={T|T < 7299041375611142144}]
I20260620 21:55:23.366549 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.362317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.17.67.195:37453)
0620 21:55:23.362453 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.362458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.362578 (+ 120us) server_negotiation.cc:407] Connection header received
0620 21:55:23.362702 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.362705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.362774 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.362911 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.363087 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.363986 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.364866 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.365258 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.365368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.365627 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.366163 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.366287 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.366350 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.366409 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:23.377640 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.374362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:42503 (local address 127.17.67.194:43899)
0620 21:55:23.374473 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.374477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.374601 (+ 124us) server_negotiation.cc:407] Connection header received
0620 21:55:23.374764 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.374768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.374820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.374898 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.375155 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.375649 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.376536 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.376754 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.376865 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.377192 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.377259 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.377335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.377426 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.377472 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:23.377655 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.374215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.17.67.195:42503)
0620 21:55:23.374475 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.374492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.374660 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.374922 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.374925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.374932 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:23.375049 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.375055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.375755 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.375761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.376403 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.376412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.376868 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.376872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.376982 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.377021 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.377350 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.377356 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.377416 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.377486 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260620 21:55:23.385165 19571 raft_consensus.cc:1261] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Refusing update from remote peer 6487b8e12442438c8336645a8abe1e60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.385340 19712 raft_consensus.cc:1261] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Refusing update from remote peer 6487b8e12442438c8336645a8abe1e60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.385820 19931 consensus_queue.cc:1035] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.386071 19928 consensus_queue.cc:1035] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.392066 19951 mvcc.cc:205] Tried to move back new op lower bound from 7299041375780491264 to 7299041374222065664. Current Snapshot: MvccSnapshot[applied={T|T < 7299041375780491264}]
I20260620 21:55:23.411157 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.406154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:49259 (local address 127.17.67.195:37453)
0620 21:55:23.406283 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.406289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.406579 (+ 290us) server_negotiation.cc:407] Connection header received
0620 21:55:23.406708 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.406710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.406749 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.406810 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.407483 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.408347 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.409752 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.410036 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.410227 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.410563 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.410651 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.410741 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.410935 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.410994 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:23.411192 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.406027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:37453 (local address 127.17.67.194:49259)
0620 21:55:23.406480 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.406495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.406611 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.406841 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.406844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.406853 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:23.407337 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.407345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.408498 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.408502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.409610 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.409622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.410236 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.410240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.410342 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.410397 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.410812 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.410818 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.410911 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.410995 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":147,"threads_started":1}
I20260620 21:55:23.438110 19415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:23.438310 19415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:23.440963 19415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Sending DeleteTablet for 3 replicas of tablet 0f7008b7d1604e259b1f01d781046293
I20260620 21:55:23.441795 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7008b7d1604e259b1f01d781046293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:23 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:38478
I20260620 21:55:23.442191 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7008b7d1604e259b1f01d781046293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:23 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60302
I20260620 21:55:23.442533 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7008b7d1604e259b1f01d781046293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:23 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 21:55:23.442632 19971 tablet_replica.cc:330] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: stopping tablet replica
I20260620 21:55:23.442983 19971 raft_consensus.cc:2229] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:23.443377 19971 raft_consensus.cc:2258] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:23.443797 19415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:23.444192 19415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:23.444676 19971 ts_tablet_manager.cc:1905] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:23.446087 19970 tablet_replica.cc:330] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: stopping tablet replica
I20260620 21:55:23.446461 19970 raft_consensus.cc:2229] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:23.446722 19972 tablet_replica.cc:330] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: stopping tablet replica
I20260620 21:55:23.446975 19970 raft_consensus.cc:2258] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:23.447283 19972 raft_consensus.cc:2229] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:23.448577 19972 raft_consensus.cc:2258] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:23.449997 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 (DEFAULT_TABLE table=test-table [id=c77e06135c3a4096885ecd46499d7c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:23.450256 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840a6cda109a4617aeda0a6ce7556ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.451458 19972 ts_tablet_manager.cc:1905] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:23.451637 19970 ts_tablet_manager.cc:1905] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:23.452960 19681 tablet_service.cc:1467] Processing CreateTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 (DEFAULT_TABLE table=test-table [id=c77e06135c3a4096885ecd46499d7c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:23.453219 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840a6cda109a4617aeda0a6ce7556ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.453882 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 (DEFAULT_TABLE table=test-table [id=c77e06135c3a4096885ecd46499d7c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:23.454154 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840a6cda109a4617aeda0a6ce7556ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:23.456534 19922 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Bootstrap starting.
I20260620 21:55:23.456787 19970 ts_tablet_manager.cc:1918] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:23.456921 19970 log.cc:1196] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/0f7008b7d1604e259b1f01d781046293
I20260620 21:55:23.457291 19970 ts_tablet_manager.cc:1939] T 0f7008b7d1604e259b1f01d781046293 P ced139d319244b3e996bc35238b35071: Deleting consensus metadata
I20260620 21:55:23.457502 19922 tablet_bootstrap.cc:654] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.457719 19923 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Bootstrap starting.
I20260620 21:55:23.458604 19923 tablet_bootstrap.cc:654] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.459625 19400 catalog_manager.cc:4841] TS ced139d319244b3e996bc35238b35071 (127.17.67.193:46805): tablet 0f7008b7d1604e259b1f01d781046293 (table test-table [id=a8cf3846941d401d8df4132bd3bf81b3]) successfully deleted
I20260620 21:55:23.461607 19921 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Bootstrap starting.
I20260620 21:55:23.462570 19921 tablet_bootstrap.cc:654] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:23.462832 19972 ts_tablet_manager.cc:1918] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:23.462962 19972 log.cc:1196] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/0f7008b7d1604e259b1f01d781046293
I20260620 21:55:23.463325 19972 ts_tablet_manager.cc:1939] T 0f7008b7d1604e259b1f01d781046293 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting consensus metadata
I20260620 21:55:23.464438 19399 catalog_manager.cc:4841] TS 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899): tablet 0f7008b7d1604e259b1f01d781046293 (table test-table [id=a8cf3846941d401d8df4132bd3bf81b3]) successfully deleted
I20260620 21:55:23.465734 19971 ts_tablet_manager.cc:1918] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:23.465833 19971 log.cc:1196] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/0f7008b7d1604e259b1f01d781046293
I20260620 21:55:23.466131 19971 ts_tablet_manager.cc:1939] T 0f7008b7d1604e259b1f01d781046293 P 6487b8e12442438c8336645a8abe1e60: Deleting consensus metadata
I20260620 21:55:23.467175 19399 catalog_manager.cc:4841] TS 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453): tablet 0f7008b7d1604e259b1f01d781046293 (table test-table [id=a8cf3846941d401d8df4132bd3bf81b3]) successfully deleted
I20260620 21:55:23.467754 19923 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: No bootstrap required, opened a new log
I20260620 21:55:23.467831 19923 ts_tablet_manager.cc:1397] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 21:55:23.468060 19921 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: No bootstrap required, opened a new log
I20260620 21:55:23.468132 19921 ts_tablet_manager.cc:1397] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 21:55:23.468276 19923 raft_consensus.cc:348] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.468382 19923 raft_consensus.cc:374] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.468412 19923 raft_consensus.cc:729] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.468516 19923 consensus_queue.cc:260] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.468564 19921 raft_consensus.cc:348] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.468679 19921 raft_consensus.cc:374] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.468712 19921 raft_consensus.cc:729] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.468775 19923 ts_tablet_manager.cc:1428] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:23.468824 19921 consensus_queue.cc:260] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.469092 19921 ts_tablet_manager.cc:1428] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:23.470659 19922 tablet_bootstrap.cc:492] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: No bootstrap required, opened a new log
I20260620 21:55:23.470770 19922 ts_tablet_manager.cc:1397] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260620 21:55:23.471235 19922 raft_consensus.cc:348] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.471367 19922 raft_consensus.cc:374] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:23.471412 19922 raft_consensus.cc:729] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced139d319244b3e996bc35238b35071, State: Initialized, Role: FOLLOWER
I20260620 21:55:23.471535 19922 consensus_queue.cc:260] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.471824 19922 ts_tablet_manager.cc:1428] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:23.656592 19929 raft_consensus.cc:482] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:23.656756 19929 raft_consensus.cc:504] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.658030 19929 leader_election.cc:288] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:23.662706 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.658350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.17.67.193:59645)
0620 21:55:23.658615 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.658630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.658803 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.659215 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.659220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.659263 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:23.659418 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.659424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.660268 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.660271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.661474 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.661484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.662338 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.662341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.662420 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.662442 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.662507 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.662562 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"spinlock_wait_cycles":29952,"thread_start_us":69,"threads_started":1}
I20260620 21:55:23.662765 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.658483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59645 (local address 127.17.67.194:43899)
0620 21:55:23.658681 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.658687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.658770 (+ 83us) server_negotiation.cc:407] Connection header received
0620 21:55:23.658945 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.658949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.659025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.659141 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:23.659542 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.660149 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.661652 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.662193 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.662318 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.662381 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.662555 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.662628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 21:55:23.663280 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840a6cda109a4617aeda0a6ce7556ec5" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" is_pre_election: true
I20260620 21:55:23.663393 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.658722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:37453 (local address 127.17.67.193:59687)
0620 21:55:23.659104 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.659118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.659269 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.659534 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.659538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.659558 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:23.660064 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.660070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.660904 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.660908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.661960 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.661967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.662965 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.662969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.663083 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.663105 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.663175 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.663243 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"spinlock_wait_cycles":8704,"thread_start_us":130,"threads_started":1}
I20260620 21:55:23.663384 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.658862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59687 (local address 127.17.67.195:37453)
0620 21:55:23.659004 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.659009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.659246 (+ 237us) server_negotiation.cc:407] Connection header received
0620 21:55:23.659374 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.659377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.659422 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.659519 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:23.660199 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.660759 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.662085 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.662733 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.662862 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.662921 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.663204 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.663273 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:55:23.663465 19712 raft_consensus.cc:2454] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 0.
I20260620 21:55:23.663867 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840a6cda109a4617aeda0a6ce7556ec5" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60" is_pre_election: true
I20260620 21:55:23.664053 19852 raft_consensus.cc:2454] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 0.
I20260620 21:55:23.664055 19503 leader_election.cc:302] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:23.664359 19929 raft_consensus.cc:2781] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:23.664417 19929 raft_consensus.cc:482] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:23.664448 19929 raft_consensus.cc:3037] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.665892 19929 raft_consensus.cc:504] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.666265 19929 leader_election.cc:288] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:23.666720 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840a6cda109a4617aeda0a6ce7556ec5" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
I20260620 21:55:23.666873 19712 raft_consensus.cc:3037] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.666860 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840a6cda109a4617aeda0a6ce7556ec5" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60"
I20260620 21:55:23.667011 19852 raft_consensus.cc:3037] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:23.668100 19712 raft_consensus.cc:2454] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 1.
I20260620 21:55:23.668126 19852 raft_consensus.cc:2454] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 1.
I20260620 21:55:23.668417 19503 leader_election.cc:302] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:23.668669 19929 raft_consensus.cc:2781] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:23.668936 19929 raft_consensus.cc:686] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 1 LEADER]: Becoming Leader. State: Replica: ced139d319244b3e996bc35238b35071, State: Running, Role: LEADER
I20260620 21:55:23.669190 19929 consensus_queue.cc:237] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:23.671859 19415 catalog_manager.cc:5495] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced139d319244b3e996bc35238b35071 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "ced139d319244b3e996bc35238b35071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:23.696966 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.693261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.17.67.193:46805)
0620 21:55:23.693400 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.693405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.693503 (+ 98us) server_negotiation.cc:407] Connection header received
0620 21:55:23.693718 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.693721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.693776 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.693886 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.694134 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.694659 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.695489 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.695718 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.695821 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.696093 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.696555 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.696704 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.696762 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.696820 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:55:23.708051 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.704301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.17.67.193:45611)
0620 21:55:23.704614 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.704628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.704727 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.704975 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.704977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.704984 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:23.705228 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.705239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.706142 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.706144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.706844 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.706852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.707383 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.707391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.707471 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.707514 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.707789 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.707794 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.707852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.707918 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":124,"threads_started":1}
I20260620 21:55:23.708039 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.704394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:45611 (local address 127.17.67.194:43899)
0620 21:55:23.704553 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.704558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.704706 (+ 148us) server_negotiation.cc:407] Connection header received
0620 21:55:23.704822 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.704824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.704874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.704974 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.705356 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.706005 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.706959 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.707248 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.707337 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.707636 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.707712 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.707769 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.707882 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.707929 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:55:23.718212 19852 raft_consensus.cc:1261] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Refusing update from remote peer ced139d319244b3e996bc35238b35071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.718358 19712 raft_consensus.cc:1261] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Refusing update from remote peer ced139d319244b3e996bc35238b35071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:23.718842 19978 consensus_queue.cc:1035] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.719723 19929 consensus_queue.cc:1035] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:23.736549 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.733555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:33221 (local address 127.17.67.193:46805)
0620 21:55:23.733672 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:23.733676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:23.733736 (+ 60us) server_negotiation.cc:407] Connection header received
0620 21:55:23.733822 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:23.733827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:23.733895 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:23.733956 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:23.734196 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.734697 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.735571 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.735801 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.735908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.736137 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.736211 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.736290 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:23.736374 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:55:23.736418 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:23.736549 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:23.733497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46805 (local address 127.17.67.194:33221)
0620 21:55:23.733650 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:23.733665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:23.733750 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:23.733986 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:23.733988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:23.733995 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:23.734096 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.734102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.734802 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.734807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.735451 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:23.735460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:23.735905 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:23.735908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:23.735990 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:23.736026 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:23.736294 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:23.736302 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:55:23.736365 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:55:23.736441 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 21:55:23.737275 19950 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:55:25.299347 19415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:25.299502 19415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:25.302208 19415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Sending DeleteTablet for 3 replicas of tablet 840a6cda109a4617aeda0a6ce7556ec5
I20260620 21:55:25.302793 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:38478
I20260620 21:55:25.302850 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 21:55:25.302903 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 840a6cda109a4617aeda0a6ce7556ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60302
I20260620 21:55:25.303164 20009 tablet_replica.cc:330] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: stopping tablet replica
I20260620 21:55:25.303324 20009 raft_consensus.cc:2229] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:25.303468 20011 tablet_replica.cc:330] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: stopping tablet replica
I20260620 21:55:25.303622 20011 raft_consensus.cc:2229] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:25.303678 20009 raft_consensus.cc:2258] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.303743 20011 raft_consensus.cc:2258] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.303941 20010 tablet_replica.cc:330] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: stopping tablet replica
I20260620 21:55:25.304083 20010 raft_consensus.cc:2229] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:25.304275 20010 raft_consensus.cc:2258] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.304513 20009 ts_tablet_manager.cc:1905] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:25.304746 20011 ts_tablet_manager.cc:1905] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:25.306301 20010 ts_tablet_manager.cc:1905] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:25.307472 20009 ts_tablet_manager.cc:1918] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:55:25.307556 20009 log.cc:1196] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/840a6cda109a4617aeda0a6ce7556ec5
I20260620 21:55:25.307709 20011 ts_tablet_manager.cc:1918] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:55:25.307850 20009 ts_tablet_manager.cc:1939] T 840a6cda109a4617aeda0a6ce7556ec5 P ced139d319244b3e996bc35238b35071: Deleting consensus metadata
I20260620 21:55:25.307881 20011 log.cc:1196] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/840a6cda109a4617aeda0a6ce7556ec5
I20260620 21:55:25.308234 20011 ts_tablet_manager.cc:1939] T 840a6cda109a4617aeda0a6ce7556ec5 P 6487b8e12442438c8336645a8abe1e60: Deleting consensus metadata
I20260620 21:55:25.308635 19400 catalog_manager.cc:4841] TS ced139d319244b3e996bc35238b35071 (127.17.67.193:46805): tablet 840a6cda109a4617aeda0a6ce7556ec5 (table test-table [id=c77e06135c3a4096885ecd46499d7c80]) successfully deleted
I20260620 21:55:25.309036 20010 ts_tablet_manager.cc:1918] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:55:25.309020 19399 catalog_manager.cc:4841] TS 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453): tablet 840a6cda109a4617aeda0a6ce7556ec5 (table test-table [id=c77e06135c3a4096885ecd46499d7c80]) successfully deleted
I20260620 21:55:25.309129 20010 log.cc:1196] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/840a6cda109a4617aeda0a6ce7556ec5
I20260620 21:55:25.309425 20010 ts_tablet_manager.cc:1939] T 840a6cda109a4617aeda0a6ce7556ec5 P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting consensus metadata
I20260620 21:55:25.310233 19399 catalog_manager.cc:4841] TS 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899): tablet 840a6cda109a4617aeda0a6ce7556ec5 (table test-table [id=c77e06135c3a4096885ecd46499d7c80]) successfully deleted
Jun 20 21:55:25 dist-test-slave-cx8d krb5kdc[19370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:55:25 dist-test-slave-cx8d krb5kdc[19370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:25.334909 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:25.323148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.17.67.254:39773)
0620 21:55:25.323436 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:25.323442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:25.323518 (+ 76us) server_negotiation.cc:407] Connection header received
0620 21:55:25.323636 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:25.323638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:25.323681 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:25.323784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:25.324457 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.325008 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.325910 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.326128 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.326206 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:25.331599 (+ 5393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:25.331602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:25.331604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:25.331630 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:25.332975 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:25.333405 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:25.333410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:25.333414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:25.333524 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:25.333879 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:25.333883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:25.333884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:25.334193 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:25.334403 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:25.334539 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:55:25.334674 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":139,"threads_started":1}
I20260620 21:55:25.337704 19415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:25.338183 19415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:25.342758 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d (DEFAULT_TABLE table=test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.342780 19681 tablet_service.cc:1467] Processing CreateTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d (DEFAULT_TABLE table=test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.343050 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215e281e2ba242b5a745ab8cf0d0b88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.343050 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215e281e2ba242b5a745ab8cf0d0b88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.346042 20021 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Bootstrap starting.
I20260620 21:55:25.346093 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d (DEFAULT_TABLE table=test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.346361 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215e281e2ba242b5a745ab8cf0d0b88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.347051 20021 tablet_bootstrap.cc:654] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.347674 20020 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Bootstrap starting.
I20260620 21:55:25.348382 20020 tablet_bootstrap.cc:654] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.348765 20023 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Bootstrap starting.
I20260620 21:55:25.349462 20020 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: No bootstrap required, opened a new log
I20260620 21:55:25.349602 20023 tablet_bootstrap.cc:654] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.349567 20020 ts_tablet_manager.cc:1397] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:25.350018 20020 raft_consensus.cc:348] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.350152 20020 raft_consensus.cc:374] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.350226 20020 raft_consensus.cc:729] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.350384 20020 consensus_queue.cc:260] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.350670 20020 ts_tablet_manager.cc:1428] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:25.350939 20023 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: No bootstrap required, opened a new log
I20260620 21:55:25.351263 20023 ts_tablet_manager.cc:1397] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 21:55:25.351712 20023 raft_consensus.cc:348] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.351816 20023 raft_consensus.cc:374] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.351845 20023 raft_consensus.cc:729] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.352057 20023 consensus_queue.cc:260] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.352427 20023 ts_tablet_manager.cc:1428] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:25.353179 20021 tablet_bootstrap.cc:492] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: No bootstrap required, opened a new log
I20260620 21:55:25.353272 20021 ts_tablet_manager.cc:1397] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 21:55:25.353763 20021 raft_consensus.cc:348] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.353873 20021 raft_consensus.cc:374] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.353915 20021 raft_consensus.cc:729] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced139d319244b3e996bc35238b35071, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.354043 20021 consensus_queue.cc:260] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.354336 20021 ts_tablet_manager.cc:1428] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:25.378400 20002 raft_consensus.cc:482] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:25.378528 20002 raft_consensus.cc:504] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.378958 20002 leader_election.cc:288] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:25.379411 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215e281e2ba242b5a745ab8cf0d0b88d" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" is_pre_election: true
I20260620 21:55:25.379433 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215e281e2ba242b5a745ab8cf0d0b88d" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60" is_pre_election: true
I20260620 21:55:25.379565 19712 raft_consensus.cc:2454] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 0.
I20260620 21:55:25.379643 19852 raft_consensus.cc:2454] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 0.
I20260620 21:55:25.379904 19503 leader_election.cc:302] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:25.380066 20002 raft_consensus.cc:2781] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:25.380138 20002 raft_consensus.cc:482] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:25.380188 20002 raft_consensus.cc:3037] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.381212 20002 raft_consensus.cc:504] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.381551 20002 leader_election.cc:288] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:25.381948 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215e281e2ba242b5a745ab8cf0d0b88d" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60"
I20260620 21:55:25.381960 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215e281e2ba242b5a745ab8cf0d0b88d" candidate_uuid: "ced139d319244b3e996bc35238b35071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b"
I20260620 21:55:25.382112 19712 raft_consensus.cc:3037] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.382120 19852 raft_consensus.cc:3037] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.383126 19852 raft_consensus.cc:2454] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 1.
I20260620 21:55:25.383198 19712 raft_consensus.cc:2454] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced139d319244b3e996bc35238b35071 in term 1.
I20260620 21:55:25.383489 19503 leader_election.cc:302] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6487b8e12442438c8336645a8abe1e60, ced139d319244b3e996bc35238b35071; no voters:
I20260620 21:55:25.383697 20002 raft_consensus.cc:2781] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:25.383800 20002 raft_consensus.cc:686] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 1 LEADER]: Becoming Leader. State: Replica: ced139d319244b3e996bc35238b35071, State: Running, Role: LEADER
I20260620 21:55:25.383945 20002 consensus_queue.cc:237] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.385815 19412 catalog_manager.cc:5495] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced139d319244b3e996bc35238b35071 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "ced139d319244b3e996bc35238b35071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:25.408136 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:25.405034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.17.67.193:46805)
0620 21:55:25.405369 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:25.405375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:25.405388 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:25.405425 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:25.405427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:25.405467 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:25.405545 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:25.405854 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.406363 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.407177 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.407392 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.407470 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:25.407708 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:25.407773 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:25.407827 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:25.407906 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 21:55:25.407946 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":162,"threads_started":1}
I20260620 21:55:25.414619 19852 raft_consensus.cc:1261] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Refusing update from remote peer ced139d319244b3e996bc35238b35071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:25.414675 19712 raft_consensus.cc:1261] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Refusing update from remote peer ced139d319244b3e996bc35238b35071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:25.415119 20002 consensus_queue.cc:1035] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:25.415246 19985 consensus_queue.cc:1035] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:25.446220 19415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:25.446376 19415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:25.448341 19415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Sending DeleteTablet for 3 replicas of tablet 215e281e2ba242b5a745ab8cf0d0b88d
I20260620 21:55:25.448910 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:38478
I20260620 21:55:25.449010 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60302
I20260620 21:55:25.449107 20009 tablet_replica.cc:330] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: stopping tablet replica
I20260620 21:55:25.449113 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet 215e281e2ba242b5a745ab8cf0d0b88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 21:55:25.449167 20011 tablet_replica.cc:330] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: stopping tablet replica
I20260620 21:55:25.449256 20009 raft_consensus.cc:2229] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:25.449259 20011 raft_consensus.cc:2229] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:25.449317 20010 tablet_replica.cc:330] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: stopping tablet replica
I20260620 21:55:25.449357 20011 raft_consensus.cc:2258] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.449419 20010 raft_consensus.cc:2229] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:25.449434 20009 raft_consensus.cc:2258] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.449551 20010 raft_consensus.cc:2258] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:25.450054 19415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:55:25.450229 20009 ts_tablet_manager.cc:1905] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:25.450305 20011 ts_tablet_manager.cc:1905] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:25.450335 20010 ts_tablet_manager.cc:1905] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:55:25.450450 19415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:25.452802 20011 ts_tablet_manager.cc:1918] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:25.452808 20009 ts_tablet_manager.cc:1918] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:25.452874 20009 log.cc:1196] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/215e281e2ba242b5a745ab8cf0d0b88d
I20260620 21:55:25.452874 20011 log.cc:1196] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/215e281e2ba242b5a745ab8cf0d0b88d
I20260620 21:55:25.452859 20010 ts_tablet_manager.cc:1918] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:25.452932 20010 log.cc:1196] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/215e281e2ba242b5a745ab8cf0d0b88d
I20260620 21:55:25.453143 20011 ts_tablet_manager.cc:1939] T 215e281e2ba242b5a745ab8cf0d0b88d P 6487b8e12442438c8336645a8abe1e60: Deleting consensus metadata
I20260620 21:55:25.453156 20009 ts_tablet_manager.cc:1939] T 215e281e2ba242b5a745ab8cf0d0b88d P ced139d319244b3e996bc35238b35071: Deleting consensus metadata
I20260620 21:55:25.453169 20010 ts_tablet_manager.cc:1939] T 215e281e2ba242b5a745ab8cf0d0b88d P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting consensus metadata
I20260620 21:55:25.454101 19400 catalog_manager.cc:4841] TS ced139d319244b3e996bc35238b35071 (127.17.67.193:46805): tablet 215e281e2ba242b5a745ab8cf0d0b88d (table test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]) successfully deleted
I20260620 21:55:25.454432 19399 catalog_manager.cc:4841] TS 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453): tablet 215e281e2ba242b5a745ab8cf0d0b88d (table test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]) successfully deleted
I20260620 21:55:25.454579 19399 catalog_manager.cc:4841] TS 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899): tablet 215e281e2ba242b5a745ab8cf0d0b88d (table test-table [id=21b6c8f662784d2ea6afdc1e143ff05a]) successfully deleted
I20260620 21:55:25.456979 19552 tablet_service.cc:1467] Processing CreateTablet for tablet d336b2676a3c455aa95a82cc8ad2371f (DEFAULT_TABLE table=test-table [id=a5386ce3f5164af9afdc3a8a5e90c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.457077 19832 tablet_service.cc:1467] Processing CreateTablet for tablet d336b2676a3c455aa95a82cc8ad2371f (DEFAULT_TABLE table=test-table [id=a5386ce3f5164af9afdc3a8a5e90c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.457062 19681 tablet_service.cc:1467] Processing CreateTablet for tablet d336b2676a3c455aa95a82cc8ad2371f (DEFAULT_TABLE table=test-table [id=a5386ce3f5164af9afdc3a8a5e90c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:25.457273 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d336b2676a3c455aa95a82cc8ad2371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.457317 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d336b2676a3c455aa95a82cc8ad2371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.457317 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d336b2676a3c455aa95a82cc8ad2371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:25.459444 20023 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Bootstrap starting.
I20260620 21:55:25.459452 20020 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Bootstrap starting.
I20260620 21:55:25.459527 20021 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: Bootstrap starting.
I20260620 21:55:25.460198 20023 tablet_bootstrap.cc:654] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.460271 20021 tablet_bootstrap.cc:654] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.460271 20020 tablet_bootstrap.cc:654] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:25.461470 20020 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: No bootstrap required, opened a new log
I20260620 21:55:25.461589 20020 ts_tablet_manager.cc:1397] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:55:25.461683 20023 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: No bootstrap required, opened a new log
I20260620 21:55:25.461750 20021 tablet_bootstrap.cc:492] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: No bootstrap required, opened a new log
I20260620 21:55:25.461768 20023 ts_tablet_manager.cc:1397] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:55:25.461830 20021 ts_tablet_manager.cc:1397] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:25.462015 20020 raft_consensus.cc:348] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462136 20020 raft_consensus.cc:374] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.462177 20020 raft_consensus.cc:729] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.462160 20023 raft_consensus.cc:348] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462280 20023 raft_consensus.cc:374] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.462270 20021 raft_consensus.cc:348] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462322 20023 raft_consensus.cc:729] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6487b8e12442438c8336645a8abe1e60, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.462308 20020 consensus_queue.cc:260] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462371 20021 raft_consensus.cc:374] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:25.462406 20021 raft_consensus.cc:729] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced139d319244b3e996bc35238b35071, State: Initialized, Role: FOLLOWER
I20260620 21:55:25.462452 20023 consensus_queue.cc:260] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462528 20021 consensus_queue.cc:260] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.462596 20020 ts_tablet_manager.cc:1428] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:25.462729 20023 ts_tablet_manager.cc:1428] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:25.462781 20021 ts_tablet_manager.cc:1428] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:25.484076 19955 raft_consensus.cc:482] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:25.484227 19955 raft_consensus.cc:504] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.484727 19955 leader_election.cc:288] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:25.485105 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d336b2676a3c455aa95a82cc8ad2371f" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60" is_pre_election: true
I20260620 21:55:25.485183 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d336b2676a3c455aa95a82cc8ad2371f" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071" is_pre_election: true
I20260620 21:55:25.485265 19852 raft_consensus.cc:2454] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 0.
I20260620 21:55:25.485399 19571 raft_consensus.cc:2454] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 0.
I20260620 21:55:25.485579 19643 leader_election.cc:302] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, 6487b8e12442438c8336645a8abe1e60; no voters:
I20260620 21:55:25.485754 19955 raft_consensus.cc:2781] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:25.485813 19955 raft_consensus.cc:482] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:25.485863 19955 raft_consensus.cc:3037] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.486806 19955 raft_consensus.cc:504] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.487146 19955 leader_election.cc:288] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 election: Requested vote from peers ced139d319244b3e996bc35238b35071 (127.17.67.193:46805), 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453)
I20260620 21:55:25.487455 19571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d336b2676a3c455aa95a82cc8ad2371f" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ced139d319244b3e996bc35238b35071"
I20260620 21:55:25.487481 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d336b2676a3c455aa95a82cc8ad2371f" candidate_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6487b8e12442438c8336645a8abe1e60"
I20260620 21:55:25.487576 19571 raft_consensus.cc:3037] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.487597 19852 raft_consensus.cc:3037] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:25.488440 19852 raft_consensus.cc:2454] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 1.
I20260620 21:55:25.488441 19571 raft_consensus.cc:2454] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b12fd0bb045ccbe4403c77816fa4b in term 1.
I20260620 21:55:25.488705 19643 leader_election.cc:302] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b12fd0bb045ccbe4403c77816fa4b, 6487b8e12442438c8336645a8abe1e60; no voters:
I20260620 21:55:25.488940 19955 raft_consensus.cc:2781] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:25.489058 19955 raft_consensus.cc:686] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 LEADER]: Becoming Leader. State: Replica: 4e9b12fd0bb045ccbe4403c77816fa4b, State: Running, Role: LEADER
I20260620 21:55:25.489235 19955 consensus_queue.cc:237] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } }
I20260620 21:55:25.490736 19412 catalog_manager.cc:5495] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194). New cstate: current_term: 1 leader_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9b12fd0bb045ccbe4403c77816fa4b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:25.511601 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:25.508249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.17.67.194:43899)
0620 21:55:25.508545 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:25.508561 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:25.508571 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:55:25.508604 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:25.508606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:25.508662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:25.508753 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:25.508998 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.509524 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.510546 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.510778 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.510885 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:25.511152 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:25.511220 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:25.511297 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:25.511393 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 21:55:25.511445 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":122,"threads_started":1}
I20260620 21:55:25.520699 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:25.517689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:49999 (local address 127.17.67.194:43899)
0620 21:55:25.517828 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:25.517832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:25.517938 (+ 106us) server_negotiation.cc:407] Connection header received
0620 21:55:25.518001 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:25.518004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:25.518042 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:25.518102 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:25.518322 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.518810 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.519600 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.519801 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.519874 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:25.520223 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:25.520303 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:25.520377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:25.520538 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 21:55:25.520582 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:55:25.520882 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:25.517526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43899 (local address 127.17.67.194:49999)
0620 21:55:25.517821 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:25.517833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:25.517918 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:25.518122 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:25.518123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:25.518129 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:25.518234 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:25.518238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.518906 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.518908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:25.519503 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:25.519511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:25.519923 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:25.519926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:25.520008 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:25.520036 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:25.520418 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:25.520424 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:25.520492 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:55:25.520557 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":147,"threads_started":1}
I20260620 21:55:25.527010 19571 raft_consensus.cc:1261] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Refusing update from remote peer 4e9b12fd0bb045ccbe4403c77816fa4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:25.526996 19852 raft_consensus.cc:1261] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Refusing update from remote peer 4e9b12fd0bb045ccbe4403c77816fa4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:25.527419 19955 consensus_queue.cc:1035] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ced139d319244b3e996bc35238b35071" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:25.527592 19994 consensus_queue.cc:1035] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6487b8e12442438c8336645a8abe1e60" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:55:25.541378 19987 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:55:27.094373 19415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:27.094522 19415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:27.096871 19415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e9ea71c1e4045f6b08522768dc6c6c2: Sending DeleteTablet for 3 replicas of tablet d336b2676a3c455aa95a82cc8ad2371f
I20260620 21:55:27.097478 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet d336b2676a3c455aa95a82cc8ad2371f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:27 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:38478
I20260620 21:55:27.097608 19681 tablet_service.cc:1514] Processing DeleteTablet for tablet d336b2676a3c455aa95a82cc8ad2371f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:27 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:33110
I20260620 21:55:27.097687 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet d336b2676a3c455aa95a82cc8ad2371f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:27 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60302
I20260620 21:55:27.097868 20063 tablet_replica.cc:330] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071: stopping tablet replica
I20260620 21:55:27.097939 20064 tablet_replica.cc:330] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: stopping tablet replica
I20260620 21:55:27.098095 20064 raft_consensus.cc:2229] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:27.098222 20063 raft_consensus.cc:2229] T d336b2676a3c455aa95a82cc8ad2371f P ced139d319244b3e996bc35238b35071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:27.098315 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19487
I20260620 21:55:27.099752 20064 raft_consensus.cc:2258] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:27.100417 20064 ts_tablet_manager.cc:1905] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:27.101787 20065 tablet_replica.cc:330] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: stopping tablet replica
I20260620 21:55:27.101941 20065 raft_consensus.cc:2229] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:27.102087 20065 raft_consensus.cc:2258] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:27.102685 20065 ts_tablet_manager.cc:1905] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:27.103930 20064 ts_tablet_manager.cc:1918] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:55:27.104019 20064 log.cc:1196] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/d336b2676a3c455aa95a82cc8ad2371f
I20260620 21:55:27.104367 20064 ts_tablet_manager.cc:1939] T d336b2676a3c455aa95a82cc8ad2371f P 4e9b12fd0bb045ccbe4403c77816fa4b: Deleting consensus metadata
I20260620 21:55:27.105160 20065 ts_tablet_manager.cc:1918] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:55:27.105190 19399 catalog_manager.cc:4841] TS 4e9b12fd0bb045ccbe4403c77816fa4b (127.17.67.194:43899): tablet d336b2676a3c455aa95a82cc8ad2371f (table test-table [id=a5386ce3f5164af9afdc3a8a5e90c90f]) successfully deleted
I20260620 21:55:27.105258 20065 log.cc:1196] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/d336b2676a3c455aa95a82cc8ad2371f
I20260620 21:55:27.105497 20065 ts_tablet_manager.cc:1939] T d336b2676a3c455aa95a82cc8ad2371f P 6487b8e12442438c8336645a8abe1e60: Deleting consensus metadata
I20260620 21:55:27.106170 19399 catalog_manager.cc:4841] TS 6487b8e12442438c8336645a8abe1e60 (127.17.67.195:37453): tablet d336b2676a3c455aa95a82cc8ad2371f (table test-table [id=a5386ce3f5164af9afdc3a8a5e90c90f]) successfully deleted
W20260620 21:55:27.107012 19400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:46805 (error 108)
I20260620 21:55:27.107141 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19627
I20260620 21:55:27.109021 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:27.108626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46805 (local address 127.0.0.1:44506)
0620 21:55:27.108861 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:27.108939 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
W20260620 21:55:27.109236 19400 catalog_manager.cc:4584] TS ced139d319244b3e996bc35238b35071 (127.17.67.193:46805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d336b2676a3c455aa95a82cc8ad2371f: Network error: Client connection negotiation failed: client connection to 127.17.67.193:46805: connect: Connection refused (error 111)
I20260620 21:55:27.114967 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19767
I20260620 21:55:27.121618 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 19383
2026-06-20T21:55:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6531 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:27 dist-test-slave-cx8d krb5kdc[20070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:27 dist-test-slave-cx8d krb5kdc[20070](info): set up 2 sockets
Jun 20 21:55:27 dist-test-slave-cx8d krb5kdc[20070](info): commencing operation
krb5kdc: starting...
W20260620 21:55:28.213449 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:28.333091 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44755
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:28.442277 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:28.442535 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:28.442575 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:28.446432 20083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:28.446487 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:28.446507 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:28.446524 20083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:28.446540 20083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:28.451614 20083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:28.452580 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:28.453521 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:28.462390 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.462464 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.462489 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:28.462432 20083 server_base.cc:1034] running on GCE node
I20260620 21:55:28.463233 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:28.464432 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:28.465687 20083 hybrid_clock.cc:648] HybridClock initialized: now 1781992528465650 us; error 47 us; skew 500 ppm
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:28.468698 20083 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:28.469732 20083 webserver.cc:460] Webserver started at http://127.17.67.254:38775/ using document root <none> and password file <none>
I20260620 21:55:28.470332 20083 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:28.470399 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:28.470664 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:28.472831 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "979ed9bd00a446d7b758a05f71426094"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "5de4c17f1c0c284c7c0b4f2c8ea473ba"
server_key_iv: "e05be43c8633ca6ea94f6fe4a5451e97"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.473444 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "979ed9bd00a446d7b758a05f71426094"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "5de4c17f1c0c284c7c0b4f2c8ea473ba"
server_key_iv: "e05be43c8633ca6ea94f6fe4a5451e97"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.477399 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 21:55:28.480073 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:28.481496 20083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:28.481695 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "979ed9bd00a446d7b758a05f71426094"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "5de4c17f1c0c284c7c0b4f2c8ea473ba"
server_key_iv: "e05be43c8633ca6ea94f6fe4a5451e97"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.481820 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:28.502821 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:28.506477 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:28.506711 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:28.514559 20083 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44755
I20260620 21:55:28.514590 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44755 every 8 connection(s)
I20260620 21:55:28.515530 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:28.518368 20151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:28.519541 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20083
I20260620 21:55:28.519675 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:28.519951 17679 external_mini_cluster.cc:1428] Setting key 77ceeb553626026656216506a48e5990
I20260620 21:55:28.524344 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094: Bootstrap starting.
I20260620 21:55:28.526520 20151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:28.527221 20151 log.cc:824] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:28.529152 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094: No bootstrap required, opened a new log
I20260620 21:55:28.533026 20151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } }
I20260620 21:55:28.533375 20151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:28.533464 20151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ed9bd00a446d7b758a05f71426094, State: Initialized, Role: FOLLOWER
I20260620 21:55:28.533960 20151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } }
I20260620 21:55:28.534118 20151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:28.534174 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:28.521365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.17.67.254:44755)
0620 21:55:28.521811 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:28.521827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:28.521862 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:55:28.522563 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:28.522581 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:28.522930 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:28.523270 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:28.524159 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.525256 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.526253 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.526810 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.526988 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:28.529469 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:28.529474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:28.529487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:28.529513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:28.531247 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:28.531678 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:28.531684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:28.531689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:28.531802 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:28.532098 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:28.532101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:28.532103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:28.532478 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:28.532708 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:28.532946 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:55:28.533226 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":155,"threads_started":1}
I20260620 21:55:28.534216 20151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:28.534338 20151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:28.535291 20151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } }
I20260620 21:55:28.535660 20151 leader_election.cc:302] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979ed9bd00a446d7b758a05f71426094; no voters:
I20260620 21:55:28.536011 20151 leader_election.cc:288] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:28.536139 20156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:28.536405 20156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [term 1 LEADER]: Becoming Leader. State: Replica: 979ed9bd00a446d7b758a05f71426094, State: Running, Role: LEADER
I20260620 21:55:28.536753 20156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } }
I20260620 21:55:28.537256 20151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:28.537827 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979ed9bd00a446d7b758a05f71426094. Latest consensus state: current_term: 1 leader_uuid: "979ed9bd00a446d7b758a05f71426094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } } }
I20260620 21:55:28.537956 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:28.538364 20161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:28.538327 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979ed9bd00a446d7b758a05f71426094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ed9bd00a446d7b758a05f71426094" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44755 } } }
I20260620 21:55:28.538461 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:28.541397 20161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:28.547420 20161 catalog_manager.cc:1347] Generated new cluster ID: dd96bde1c09043c48ebf0b8a4dfb22a4
I20260620 21:55:28.547497 20161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:28.557055 20161 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:28.558044 20161 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:28.568235 20161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979ed9bd00a446d7b758a05f71426094: Generated new TSK 0
I20260620 21:55:28.569079 20161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:28.603937 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:28.714448 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:28.714707 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:28.714784 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:28.718961 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:28.719022 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:28.719118 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:28.724431 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:28.725766 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:28.727205 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:28.738421 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.738446 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.738433 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:28.739084 20177 server_base.cc:1034] running on GCE node
I20260620 21:55:28.739504 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:28.740186 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:28.741370 20177 hybrid_clock.cc:648] HybridClock initialized: now 1781992528741333 us; error 49 us; skew 500 ppm
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:28.744221 20177 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:28.745172 20177 webserver.cc:460] Webserver started at http://127.17.67.193:39841/ using document root <none> and password file <none>
I20260620 21:55:28.745801 20177 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:28.745891 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:28.746129 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:28.748205 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "945b6a96cea94013a8d96e990f71c111"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "4e4c3d6272ff7c40d4476b02f3b265ba"
server_key_iv: "afd170490a687f1b94b3ef668af56899"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.748770 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "945b6a96cea94013a8d96e990f71c111"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "4e4c3d6272ff7c40d4476b02f3b265ba"
server_key_iv: "afd170490a687f1b94b3ef668af56899"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.752985 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:55:28.755640 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:28.756842 20177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:28.756978 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "945b6a96cea94013a8d96e990f71c111"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "4e4c3d6272ff7c40d4476b02f3b265ba"
server_key_iv: "afd170490a687f1b94b3ef668af56899"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.757128 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:28.768573 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:28.771847 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:28.772054 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:28.772641 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:28.773483 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:28.773559 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:28.773694 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:28.773746 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:28.783113 20177 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:33033
I20260620 21:55:28.783141 20305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:33033 every 8 connection(s)
I20260620 21:55:28.784024 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:28.790282 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20177
I20260620 21:55:28.790407 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:28.790704 17679 external_mini_cluster.cc:1428] Setting key 6466174858d5566afe6d4128d9984f90
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:28.799813 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:28.785875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:50349 (local address 127.17.67.254:44755)
0620 21:55:28.786012 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:28.786064 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:28.786924 (+ 860us) server_negotiation.cc:407] Connection header received
0620 21:55:28.788397 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:28.788401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:28.788465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:28.788586 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:28.789923 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.790838 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.792189 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.792530 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.792656 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:28.795605 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:28.795613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:28.795620 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:28.795658 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:28.797542 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:28.798071 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:28.798078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:28.798082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:28.798185 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:28.798516 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:28.798519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:28.798524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:28.798818 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:28.798937 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:28.799457 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 21:55:28.799600 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:28.800578 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:28.786285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44755 (local address 127.17.67.193:50349)
0620 21:55:28.786769 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:28.786813 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:28.788168 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:28.788761 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:28.788768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:28.789152 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:28.789767 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:28.789784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.790987 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.790990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:28.792042 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:28.792056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:28.792693 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:28.792696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:28.792845 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:28.793475 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:28.793492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:28.795428 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:28.797682 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:28.797688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:28.797702 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:28.797959 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:28.798283 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:28.798285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:28.798286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:28.798417 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:28.798948 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:28.798954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:28.799207 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 21:55:28.799369 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:55:28.799643 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:28.802026 20306 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44755
I20260620 21:55:28.802248 20306 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:28.802724 20306 heartbeater.cc:511] Master 127.17.67.254:44755 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:55:28.804533 20115 ts_manager.cc:194] Registered new tserver with Master: 945b6a96cea94013a8d96e990f71c111 (127.17.67.193:33033)
I20260620 21:55:28.806059 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:50349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:28.820778 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:28.928799 20311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:28.929105 20311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:28.929198 20311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:28.933234 20311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:28.933324 20311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:28.933416 20311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:28.938818 20311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:28.939884 20311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:28.940879 20311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:55:28.951109 20311 server_base.cc:1034] running on GCE node
W20260620 21:55:28.951069 20316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.951156 20319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:28.951094 20317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:28.951745 20311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:28.952426 20311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:28.953670 20311 hybrid_clock.cc:648] HybridClock initialized: now 1781992528953647 us; error 33 us; skew 500 ppm
Jun 20 21:55:28 dist-test-slave-cx8d krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:28.956666 20311 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:28.957708 20311 webserver.cc:460] Webserver started at http://127.17.67.194:32917/ using document root <none> and password file <none>
I20260620 21:55:28.958307 20311 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:28.958374 20311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:28.958592 20311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:28.960726 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "64dc469e47b14d4bb5738b67f3b2eb78"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "ee3ce8f0bb1858bde2cb79ccb5961112"
server_key_iv: "dd505a100746d1d22960f8cff961dc90"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.961395 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "64dc469e47b14d4bb5738b67f3b2eb78"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "ee3ce8f0bb1858bde2cb79ccb5961112"
server_key_iv: "dd505a100746d1d22960f8cff961dc90"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.965196 20311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:55:28.967824 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:28.969255 20311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:28.969350 20311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "64dc469e47b14d4bb5738b67f3b2eb78"
format_stamp: "Formatted at 2026-06-20 21:55:28 on dist-test-slave-cx8d"
server_key: "ee3ce8f0bb1858bde2cb79ccb5961112"
server_key_iv: "dd505a100746d1d22960f8cff961dc90"
server_key_version: "encryptionkey@0"
I20260620 21:55:28.969489 20311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:29.005511 20311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:29.008805 20311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:29.009024 20311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:29.009873 20311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:29.010782 20311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:29.010838 20311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:29.010906 20311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:29.010951 20311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:29.020291 20311 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:38757
I20260620 21:55:29.020313 20439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:38757 every 8 connection(s)
I20260620 21:55:29.021250 20311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:29.027634 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20311
I20260620 21:55:29.027758 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:29.028045 17679 external_mini_cluster.cc:1428] Setting key c416c2da91327297c8e153e69fbc3b38
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992528, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:29.036182 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:29.023143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:37759 (local address 127.17.67.254:44755)
0620 21:55:29.023292 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:29.023295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:29.024213 (+ 918us) server_negotiation.cc:407] Connection header received
0620 21:55:29.025547 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:29.025550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:29.025636 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:29.025710 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:29.026843 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.027402 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.028886 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.029104 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.029188 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:29.032028 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:29.032031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:29.032033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:29.032057 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:29.033639 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:29.034303 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:29.034306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:29.034308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:29.034386 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:29.034810 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:29.034816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:29.034820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:29.035051 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:29.035134 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:29.035913 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 21:55:29.036029 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:55:29.037410 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:29.023533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44755 (local address 127.17.67.194:37759)
0620 21:55:29.024033 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:29.024078 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:29.025335 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:29.025858 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:29.025865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:29.026235 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:29.026718 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:29.026731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.027529 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.027532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:29.028749 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:29.028759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.029263 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.029268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:29.029427 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:29.030074 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:29.030099 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:29.031849 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:29.033751 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:29.033760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:29.033781 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:29.034184 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:29.034479 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:29.034482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:29.034484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:29.034685 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:29.035172 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:29.035181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:29.035562 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 21:55:29.035819 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 21:55:29.036230 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:29.038755 20440 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44755
I20260620 21:55:29.039000 20440 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:29.039597 20440 heartbeater.cc:511] Master 127.17.67.254:44755 requested a full tablet report, sending...
I20260620 21:55:29.040771 20115 ts_manager.cc:194] Registered new tserver with Master: 64dc469e47b14d4bb5738b67f3b2eb78 (127.17.67.194:38757)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:55:29.041432 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:37759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:29.055747 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:29.172854 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:29.173115 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:29.173158 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:29.177268 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:29.177338 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:29.177434 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:29.182698 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:29.183738 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:29.184777 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:29.194089 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:29.194113 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:29.194103 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:29.194399 20445 server_base.cc:1034] running on GCE node
I20260620 21:55:29.194878 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:29.195531 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:29.196724 20445 hybrid_clock.cc:648] HybridClock initialized: now 1781992529196711 us; error 26 us; skew 500 ppm
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992529, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:29.199752 20445 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:29.200778 20445 webserver.cc:460] Webserver started at http://127.17.67.195:36697/ using document root <none> and password file <none>
I20260620 21:55:29.201341 20445 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:29.201406 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:29.201656 20445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:29.203893 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "d7047dc2b92b42068404dad6746bc5bc"
format_stamp: "Formatted at 2026-06-20 21:55:29 on dist-test-slave-cx8d"
server_key: "5da39f20810eb44722226953df7225b5"
server_key_iv: "d14edb91e96c7119404b268bfdf77b94"
server_key_version: "encryptionkey@0"
I20260620 21:55:29.204455 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "d7047dc2b92b42068404dad6746bc5bc"
format_stamp: "Formatted at 2026-06-20 21:55:29 on dist-test-slave-cx8d"
server_key: "5da39f20810eb44722226953df7225b5"
server_key_iv: "d14edb91e96c7119404b268bfdf77b94"
server_key_version: "encryptionkey@0"
I20260620 21:55:29.208468 20445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:29.211232 20460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:29.212608 20445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:29.212756 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "d7047dc2b92b42068404dad6746bc5bc"
format_stamp: "Formatted at 2026-06-20 21:55:29 on dist-test-slave-cx8d"
server_key: "5da39f20810eb44722226953df7225b5"
server_key_iv: "d14edb91e96c7119404b268bfdf77b94"
server_key_version: "encryptionkey@0"
I20260620 21:55:29.212885 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:29.242653 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:29.245911 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:29.246145 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:29.246785 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:29.247671 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:29.247749 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:29.247828 20445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:29.247885 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:29.257928 20445 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:41283
I20260620 21:55:29.257995 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:41283 every 8 connection(s)
I20260620 21:55:29.258900 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:29.262849 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20445
I20260620 21:55:29.262974 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:29.263303 17679 external_mini_cluster.cc:1428] Setting key 7789b50aab249e6d08084379f5580f9f
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992529, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:29.273799 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:29.260854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:41217 (local address 127.17.67.254:44755)
0620 21:55:29.260987 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:29.260991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:29.262028 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 21:55:29.263751 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:29.263756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:29.263832 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:29.263921 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:29.265515 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.266149 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.267386 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.267606 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.267683 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:29.270168 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:29.270171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:29.270174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:29.270200 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:29.271684 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:29.272196 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:29.272201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:29.272203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:29.272283 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:29.272621 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:29.272626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:29.272628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:29.272876 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:29.272989 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:29.273479 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 21:55:29.273629 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:29.274694 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:29.261350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44755 (local address 127.17.67.195:41217)
0620 21:55:29.261872 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:29.261921 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:29.263464 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:29.264130 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:29.264138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:29.264713 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:29.265320 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:29.265343 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.266276 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.266278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:29.267269 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:29.267278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:29.267746 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:29.267749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:29.267902 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:29.268477 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:29.268494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:29.270003 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:29.271816 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:29.271822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:29.271836 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:29.272092 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:29.272374 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:29.272378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:29.272381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:29.272519 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:29.272989 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:29.272997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:29.273250 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 21:55:29.273440 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 21:55:29.273804 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260620 21:55:29.275826 20574 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44755
I20260620 21:55:29.276139 20574 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:29.276778 20574 heartbeater.cc:511] Master 127.17.67.254:44755 requested a full tablet report, sending...
I20260620 21:55:29.277851 20115 ts_manager.cc:194] Registered new tserver with Master: d7047dc2b92b42068404dad6746bc5bc (127.17.67.195:41283)
I20260620 21:55:29.278441 20115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:41217
I20260620 21:55:29.289362 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:55:29.302673 20154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:29.301112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.17.67.254:44755)
0620 21:55:29.301276 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:29.301280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:29.301396 (+ 116us) server_negotiation.cc:407] Connection header received
0620 21:55:29.301512 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:29.301515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:29.301559 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:29.301683 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:29.302522 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37362: BlockingRecv error: recv got EOF from 127.0.0.1:37362 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 21:55:29.303793 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20177
I20260620 21:55:29.309303 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20311
I20260620 21:55:29.314690 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20445
I20260620 21:55:29.320515 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20083
2026-06-20T21:55:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2198 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20586](info): set up 2 sockets
Jun 20 21:55:29 dist-test-slave-cx8d krb5kdc[20586](info): commencing operation
krb5kdc: starting...
W20260620 21:55:30.422322 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:30.545639 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43205
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:30.655246 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:30.655571 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:30.655645 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:30.659713 20599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:30.659801 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:30.659840 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:30.659893 20599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:30.659931 20599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:30.665171 20599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:30.666240 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:30.667273 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:30.675565 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:30.675657 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:30.675550 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:30.675894 20599 server_base.cc:1034] running on GCE node
I20260620 21:55:30.676570 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:30.677807 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:30.679037 20599 hybrid_clock.cc:648] HybridClock initialized: now 1781992530679008 us; error 41 us; skew 500 ppm
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992530, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:30.687017 20599 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:30.687958 20599 webserver.cc:460] Webserver started at http://127.17.67.254:37849/ using document root <none> and password file <none>
I20260620 21:55:30.688560 20599 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:30.688623 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:30.688796 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:30.691001 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "cf3f7483bef44faaab42fd1522d3b779"
format_stamp: "Formatted at 2026-06-20 21:55:30 on dist-test-slave-cx8d"
server_key: "436eead622a46166f599f49d4a70ae2d"
server_key_iv: "8b3ca61d6cd8f95ec9eecf39fa224b7d"
server_key_version: "encryptionkey@0"
I20260620 21:55:30.691545 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "cf3f7483bef44faaab42fd1522d3b779"
format_stamp: "Formatted at 2026-06-20 21:55:30 on dist-test-slave-cx8d"
server_key: "436eead622a46166f599f49d4a70ae2d"
server_key_iv: "8b3ca61d6cd8f95ec9eecf39fa224b7d"
server_key_version: "encryptionkey@0"
I20260620 21:55:30.695490 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:30.698001 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:30.699285 20599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:30.699424 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "cf3f7483bef44faaab42fd1522d3b779"
format_stamp: "Formatted at 2026-06-20 21:55:30 on dist-test-slave-cx8d"
server_key: "436eead622a46166f599f49d4a70ae2d"
server_key_iv: "8b3ca61d6cd8f95ec9eecf39fa224b7d"
server_key_version: "encryptionkey@0"
I20260620 21:55:30.699533 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:30.715948 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:30.719349 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:30.719552 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:30.726528 20599 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:43205
I20260620 21:55:30.726823 20666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:43205 every 8 connection(s)
I20260620 21:55:30.727514 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:30.730451 20667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:30.732448 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20599
I20260620 21:55:30.732587 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:30.732858 17679 external_mini_cluster.cc:1428] Setting key 6944c0fc088e4b4cdfb3deb7605a8407
I20260620 21:55:30.737203 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: Bootstrap starting.
I20260620 21:55:30.739795 20667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:30.740743 20667 log.cc:824] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:30.743016 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: No bootstrap required, opened a new log
I20260620 21:55:30.746716 20667 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } }
I20260620 21:55:30.746990 20667 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:30.747072 20667 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3f7483bef44faaab42fd1522d3b779, State: Initialized, Role: FOLLOWER
I20260620 21:55:30.747567 20667 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } }
I20260620 21:55:30.747731 20667 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:30.747820 20667 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:30.747949 20667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:30.748334 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:30.734198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.17.67.254:43205)
0620 21:55:30.734609 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:30.734626 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:30.734660 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:55:30.735295 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:30.735311 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:30.735652 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:30.735931 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:30.736969 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:30.738219 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:30.739386 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:30.739888 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:30.740121 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:30.742999 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:30.743004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:30.743018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:30.743047 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:30.744985 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:30.745490 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:30.745496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:30.745504 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:30.745692 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:30.746019 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:30.746024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:30.746026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:30.746453 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:30.746690 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:30.746986 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:55:30.747295 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260620 21:55:30.748989 20667 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } }
I20260620 21:55:30.749356 20667 leader_election.cc:302] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3f7483bef44faaab42fd1522d3b779; no voters:
I20260620 21:55:30.749729 20667 leader_election.cc:288] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:30.749852 20672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:30.750113 20672 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [term 1 LEADER]: Becoming Leader. State: Replica: cf3f7483bef44faaab42fd1522d3b779, State: Running, Role: LEADER
I20260620 21:55:30.750485 20672 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } }
I20260620 21:55:30.751061 20667 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:30.751715 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:30.753996 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3f7483bef44faaab42fd1522d3b779. Latest consensus state: current_term: 1 leader_uuid: "cf3f7483bef44faaab42fd1522d3b779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } } }
I20260620 21:55:30.754200 20674 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3f7483bef44faaab42fd1522d3b779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3f7483bef44faaab42fd1522d3b779" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43205 } } }
I20260620 21:55:30.754232 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:30.754515 20674 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779 [sys.catalog]: This master's current role is: LEADER
W20260620 21:55:30.756179 20686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:30.756278 20686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:30.756390 20689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:30.757210 20689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:30.763165 20689 catalog_manager.cc:1347] Generated new cluster ID: a5536d37aae1462b87d4fdd3c54f57bf
I20260620 21:55:30.763240 20689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:30.773622 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:30.767566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.17.67.254:43205)
0620 21:55:30.767690 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:30.767695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:30.767988 (+ 293us) server_negotiation.cc:407] Connection header received
0620 21:55:30.768124 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:30.768128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:30.768189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:30.768287 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:30.768929 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:30.769764 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:30.771039 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:30.771408 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:30.771514 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:30.772488 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:30.772492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:30.772495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:30.773105 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:30.773141 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:30.773161 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:30.773255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:30.773376 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 21:55:30.773441 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:30.806766 20689 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:30.808210 20689 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:30.823671 20689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf3f7483bef44faaab42fd1522d3b779: Generated new TSK 0
I20260620 21:55:30.824489 20689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:30.838621 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20599
2026-06-20T21:55:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1514 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20700](info): set up 2 sockets
Jun 20 21:55:30 dist-test-slave-cx8d krb5kdc[20700](info): commencing operation
krb5kdc: starting...
W20260620 21:55:31.897538 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:31 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:32.018518 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:34777
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:32.129783 20713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:32.130115 20713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:32.130203 20713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:32.134316 20713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:32.134414 20713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:32.134454 20713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:32.134500 20713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:32.134538 20713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:32.140071 20713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:32.141238 20713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:32.142351 20713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:32.151034 20718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.151039 20721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.151261 20719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.151290 20713 server_base.cc:1034] running on GCE node
I20260620 21:55:32.151935 20713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:32.153048 20713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:32.154217 20713 hybrid_clock.cc:648] HybridClock initialized: now 1781992532154196 us; error 34 us; skew 500 ppm
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:32.157091 20713 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:32.158070 20713 webserver.cc:460] Webserver started at http://127.17.67.254:40697/ using document root <none> and password file <none>
I20260620 21:55:32.158638 20713 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:32.158727 20713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:32.158951 20713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:32.160928 20713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "8a7030eadd8843b89d6ebdff1ae667d6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "d46eb053d00fbaaffb21ae8e43b6e254"
server_key_iv: "38d008d4fff3227347b682d53b7ced6b"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.161497 20713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "8a7030eadd8843b89d6ebdff1ae667d6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "d46eb053d00fbaaffb21ae8e43b6e254"
server_key_iv: "38d008d4fff3227347b682d53b7ced6b"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.165122 20713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:32.167552 20728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.168843 20713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:32.169030 20713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "8a7030eadd8843b89d6ebdff1ae667d6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "d46eb053d00fbaaffb21ae8e43b6e254"
server_key_iv: "38d008d4fff3227347b682d53b7ced6b"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.169154 20713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:32.192472 20713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:32.195883 20713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:32.196102 20713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:32.203383 20713 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:34777
I20260620 21:55:32.203698 20780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:34777 every 8 connection(s)
I20260620 21:55:32.204387 20713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:32.204933 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20713
I20260620 21:55:32.205086 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:32.205384 17679 external_mini_cluster.cc:1428] Setting key fe449a79fa259085d10b84a4699cc87e
I20260620 21:55:32.207620 20781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:32.214403 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: Bootstrap starting.
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:32.217022 20781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:32.217952 20781 log.cc:824] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:32.219955 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: No bootstrap required, opened a new log
I20260620 21:55:32.220625 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.206894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.17.67.254:34777)
0620 21:55:32.207284 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:32.207305 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:32.207350 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:55:32.208188 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:32.208210 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:32.208622 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:32.208991 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:32.209755 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.211295 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.212211 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.212894 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.213070 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.215695 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:32.215701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:32.215717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:32.215748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:32.217749 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.218205 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.218211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.218217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.218361 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.218684 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.218689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.218691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.219080 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:32.219281 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:32.219572 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 21:55:32.219858 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260620 21:55:32.223611 20781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } }
I20260620 21:55:32.223851 20781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:32.223935 20781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7030eadd8843b89d6ebdff1ae667d6, State: Initialized, Role: FOLLOWER
I20260620 21:55:32.224359 20781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } }
I20260620 21:55:32.224505 20781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:32.224596 20781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:32.224714 20781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:32.225699 20781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } }
I20260620 21:55:32.226035 20781 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7030eadd8843b89d6ebdff1ae667d6; no voters:
I20260620 21:55:32.226379 20781 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:32.226568 20786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:32.226863 20786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7030eadd8843b89d6ebdff1ae667d6, State: Running, Role: LEADER
I20260620 21:55:32.227275 20786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } }
I20260620 21:55:32.227734 20781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:32.229743 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } } }
I20260620 21:55:32.229780 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7030eadd8843b89d6ebdff1ae667d6. Latest consensus state: current_term: 1 leader_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7030eadd8843b89d6ebdff1ae667d6" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34777 } } }
I20260620 21:55:32.229877 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:32.229866 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6 [sys.catalog]: This master's current role is: LEADER
W20260620 21:55:32.232654 20800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:32.232705 20800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:32.232785 20801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:32.233676 20801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:32.239164 20801 catalog_manager.cc:1347] Generated new cluster ID: 79b061ce901141f885a7c95be2f1afdc
I20260620 21:55:32.239243 20801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:32.268615 20801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:32.269611 20801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:32.277009 20801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7030eadd8843b89d6ebdff1ae667d6: Generated new TSK 0
I20260620 21:55:32.277935 20801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:32.318392 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:32.449509 20807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:32.449782 20807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:32.449824 20807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:32.453934 20807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:32.453992 20807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:32.454051 20807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:32.459136 20807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:32.460088 20807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:32.461023 20807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:32.470059 20815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.470139 20813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.470063 20812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.470476 20807 server_base.cc:1034] running on GCE node
I20260620 21:55:32.470883 20807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:32.471491 20807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:32.472724 20807 hybrid_clock.cc:648] HybridClock initialized: now 1781992532472706 us; error 36 us; skew 500 ppm
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:32.475658 20807 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:32.476584 20807 webserver.cc:460] Webserver started at http://127.17.67.193:44425/ using document root <none> and password file <none>
I20260620 21:55:32.477142 20807 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:32.477240 20807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:32.477474 20807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:32.479534 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "a4fc7f15e23b46d38b7c932059c27d23"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "6920208aab711390539ea9a552b3c869"
server_key_iv: "d3cc5c7aa0f644745055cabfe587404c"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.480290 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "a4fc7f15e23b46d38b7c932059c27d23"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "6920208aab711390539ea9a552b3c869"
server_key_iv: "d3cc5c7aa0f644745055cabfe587404c"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.484084 20807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:55:32.486476 20822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.487702 20807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:32.487848 20807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "a4fc7f15e23b46d38b7c932059c27d23"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "6920208aab711390539ea9a552b3c869"
server_key_iv: "d3cc5c7aa0f644745055cabfe587404c"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.487951 20807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:32.511014 20807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:32.514357 20807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:32.514590 20807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:32.515219 20807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:32.516089 20807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:32.516140 20807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.516183 20807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:32.516234 20807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.525750 20807 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:42197
I20260620 21:55:32.525779 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:42197 every 8 connection(s)
I20260620 21:55:32.526698 20807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:32.535651 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20807
I20260620 21:55:32.535799 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:32.536155 17679 external_mini_cluster.cc:1428] Setting key 430a0aa0815b39ba79b4838f7899e243
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:32.542246 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.528658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:55805 (local address 127.17.67.254:34777)
0620 21:55:32.528784 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:32.528788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:32.529827 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 21:55:32.531358 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:32.531361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:32.531426 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:32.531513 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:32.532737 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.533327 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.534675 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.534941 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.535068 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.538000 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:32.538005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:32.538010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:32.538041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:32.539821 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.540506 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.540509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.540510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.540608 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.541019 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.541022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.541023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.541207 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:32.541288 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:32.542004 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 21:55:32.542118 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:55:32.543555 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.529144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:34777 (local address 127.17.67.193:55805)
0620 21:55:32.529676 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:32.529722 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:32.531131 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:32.531691 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:32.531699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:32.532089 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:32.532577 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.532591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.533459 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.533463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.534517 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.534527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.535110 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.535114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.535297 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.535984 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:32.536008 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:32.537825 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:32.539955 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.539962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.539980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.540370 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.540705 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.540710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.540714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.540916 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.541318 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:32.541328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:32.541720 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 21:55:32.541936 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 21:55:32.542336 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260620 21:55:32.544943 20936 heartbeater.cc:347] Connected to a master server at 127.17.67.254:34777
I20260620 21:55:32.545202 20936 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:32.545775 20936 heartbeater.cc:511] Master 127.17.67.254:34777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:32.547470 20745 ts_manager.cc:194] Registered new tserver with Master: a4fc7f15e23b46d38b7c932059c27d23 (127.17.67.193:42197)
I20260620 21:55:32.548826 20745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:55805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:32.565171 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:32.678252 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:32.678566 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:32.678653 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:32.683125 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:32.683209 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:32.683300 20941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:32.688472 20941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:32.689579 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:32.690687 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:32.700642 20947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.700736 20946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.700868 20941 server_base.cc:1034] running on GCE node
W20260620 21:55:32.700652 20949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.701404 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:32.702111 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:32.703310 20941 hybrid_clock.cc:648] HybridClock initialized: now 1781992532703272 us; error 49 us; skew 500 ppm
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:32.706265 20941 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:32.707165 20941 webserver.cc:460] Webserver started at http://127.17.67.194:35397/ using document root <none> and password file <none>
I20260620 21:55:32.707739 20941 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:32.707830 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:32.708071 20941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:32.710201 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "11835829f95247d89fa2349efa11fef6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "b26e6aff9d6ecedfc75b1bf68c8ef834"
server_key_iv: "2b9fbbe86291a355b912da952afa43c7"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.710826 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "11835829f95247d89fa2349efa11fef6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "b26e6aff9d6ecedfc75b1bf68c8ef834"
server_key_iv: "2b9fbbe86291a355b912da952afa43c7"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.714660 20941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 21:55:32.717021 20956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.718313 20941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:32.718406 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "11835829f95247d89fa2349efa11fef6"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "b26e6aff9d6ecedfc75b1bf68c8ef834"
server_key_iv: "2b9fbbe86291a355b912da952afa43c7"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.718487 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:32.727421 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:32.730840 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:32.731001 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:32.731559 20941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:32.732292 20941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:32.732337 20941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.732374 20941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:32.732389 20941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.741317 20941 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:44435
I20260620 21:55:32.741358 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:44435 every 8 connection(s)
I20260620 21:55:32.742259 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:32.751660 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 20941
I20260620 21:55:32.751756 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:32.752061 17679 external_mini_cluster.cc:1428] Setting key 984440d5b744e4f5ed7131dca6a4d21e
I20260620 21:55:32.756841 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.744141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:41535 (local address 127.17.67.254:34777)
0620 21:55:32.744276 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:32.744279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:32.745023 (+ 744us) server_negotiation.cc:407] Connection header received
0620 21:55:32.746452 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:32.746456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:32.746515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:32.746628 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:32.747764 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.748293 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.749463 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.749784 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.749898 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.752676 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:32.752679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:32.752682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:32.752710 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:32.754343 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.755027 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.755030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.755031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.755116 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.755562 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.755568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.755570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.755817 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:32.755924 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:32.756459 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 21:55:32.756595 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:32.757929 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.744428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:34777 (local address 127.17.67.194:41535)
0620 21:55:32.744875 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:32.744916 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:32.746244 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:32.746802 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:32.746809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:32.747152 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:32.747618 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.747630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.748419 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.748423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.749334 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.749343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.749935 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.749939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.750107 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.750671 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:32.750687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:32.752462 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:32.754466 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.754475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.754494 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.754900 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.755241 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.755248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.755253 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.755463 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.755942 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:32.755948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:32.756205 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 21:55:32.756376 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 21:55:32.756672 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260620 21:55:32.759188 21070 heartbeater.cc:347] Connected to a master server at 127.17.67.254:34777
I20260620 21:55:32.759452 21070 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:32.759989 21070 heartbeater.cc:511] Master 127.17.67.254:34777 requested a full tablet report, sending...
I20260620 21:55:32.761165 20745 ts_manager.cc:194] Registered new tserver with Master: 11835829f95247d89fa2349efa11fef6 (127.17.67.194:44435)
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:32.761852 20745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:41535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:32.780447 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:32.890187 21075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:32.890486 21075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:32.890564 21075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:32.894735 21075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:32.894819 21075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:32.894917 21075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:32.900133 21075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:32.901187 21075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:32.902264 21075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:32.912130 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:32.912133 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.912393 21075 server_base.cc:1034] running on GCE node
W20260620 21:55:32.912130 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:32.912938 21075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:32.913655 21075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:32.914868 21075 hybrid_clock.cc:648] HybridClock initialized: now 1781992532914840 us; error 42 us; skew 500 ppm
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:32.918000 21075 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:32.919051 21075 webserver.cc:460] Webserver started at http://127.17.67.195:33679/ using document root <none> and password file <none>
I20260620 21:55:32.919592 21075 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:32.919656 21075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:32.919822 21075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:32.922001 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "f9523ea2289247729ec02a23c7971bec"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "e6f18d61e85ffc36bd9d99c6d2534550"
server_key_iv: "c91daffecd98bf4b63dee09acb55e8bd"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.922657 21075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "f9523ea2289247729ec02a23c7971bec"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "e6f18d61e85ffc36bd9d99c6d2534550"
server_key_iv: "c91daffecd98bf4b63dee09acb55e8bd"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.926540 21075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:32.929692 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.931110 21075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:32.931249 21075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "f9523ea2289247729ec02a23c7971bec"
format_stamp: "Formatted at 2026-06-20 21:55:32 on dist-test-slave-cx8d"
server_key: "e6f18d61e85ffc36bd9d99c6d2534550"
server_key_iv: "c91daffecd98bf4b63dee09acb55e8bd"
server_key_version: "encryptionkey@0"
I20260620 21:55:32.931386 21075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:32.947306 21075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:32.951099 21075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:32.951340 21075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:32.952010 21075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:32.952872 21075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:32.952930 21075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.952975 21075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:32.952992 21075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:32.962409 21075 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:42085
I20260620 21:55:32.962486 21203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:42085 every 8 connection(s)
I20260620 21:55:32.963366 21075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:32.966991 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21075
I20260620 21:55:32.967128 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:32.967432 17679 external_mini_cluster.cc:1428] Setting key ccdba74bc275d61c97b7b3ecf8796f7a
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:32.978138 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.965227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:50433 (local address 127.17.67.254:34777)
0620 21:55:32.965366 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:32.965370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:32.966272 (+ 902us) server_negotiation.cc:407] Connection header received
0620 21:55:32.967952 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:32.967957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:32.968027 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:32.968132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:32.969372 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.970005 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.971222 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.971473 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.971570 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.974448 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:32.974451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:32.974454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:32.974484 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:32.975940 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.976434 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.976437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.976438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.976518 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:32.976814 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:32.976816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:32.976818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:32.977032 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:32.977114 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:32.977787 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 21:55:32.977991 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:32.978999 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:32.965659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:34777 (local address 127.17.67.195:50433)
0620 21:55:32.966119 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:32.966163 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:32.967683 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:32.968322 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:32.968329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:32.968712 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:32.969213 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.969227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.970116 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.970119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.971081 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:32.971090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:32.971613 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:32.971616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:32.971757 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:32.972415 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:32.972434 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:32.974270 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:32.976060 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.976066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.976082 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.976331 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.976602 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:32.976604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:32.976605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:32.976733 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:32.977159 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:32.977165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:32.977413 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 21:55:32.977728 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 21:55:32.978106 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260620 21:55:32.980341 21204 heartbeater.cc:347] Connected to a master server at 127.17.67.254:34777
I20260620 21:55:32.980615 21204 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:32.981310 21204 heartbeater.cc:511] Master 127.17.67.254:34777 requested a full tablet report, sending...
I20260620 21:55:32.982353 20745 ts_manager.cc:194] Registered new tserver with Master: f9523ea2289247729ec02a23c7971bec (127.17.67.195:42085)
I20260620 21:55:32.982933 20745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:50433
I20260620 21:55:32.993363 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:55:32 dist-test-slave-cx8d krb5kdc[20700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 21:55:33 dist-test-slave-cx8d krb5kdc[20700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992532, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:33.020674 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:33.011558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.17.67.254:34777)
0620 21:55:33.011752 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:33.011757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:33.011959 (+ 202us) server_negotiation.cc:407] Connection header received
0620 21:55:33.012105 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:33.012109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:33.012154 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:33.012244 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:33.012982 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:33.013765 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:33.014632 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:33.014848 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:33.014956 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:33.017420 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:33.017424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:33.017426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:33.017449 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:33.018948 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:33.019389 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:33.019394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:33.019396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:33.019479 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:33.019760 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:33.019764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:33.019765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:33.019992 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:33.020141 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:33.020384 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 21:55:33.020501 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 21:55:33.021191 20745 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46532
I20260620 21:55:33.022156 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20807
I20260620 21:55:33.028179 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20941
I20260620 21:55:33.033730 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21075
I20260620 21:55:33.038803 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 20713
2026-06-20T21:55:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2203 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:33 dist-test-slave-cx8d krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:33 dist-test-slave-cx8d krb5kdc[21216](info): set up 2 sockets
Jun 20 21:55:33 dist-test-slave-cx8d krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260620 21:55:34.098294 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:34.217672 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:40841
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:34.327699 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:34.328011 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:34.328097 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:34.332152 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:34.332249 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:34.332288 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:34.332325 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:34.332362 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:34.337776 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:40841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:34.338853 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:34.339890 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:34.348402 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.348407 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.348364 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:34.348645 21229 server_base.cc:1034] running on GCE node
I20260620 21:55:34.349205 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:34.350355 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:34.351521 21229 hybrid_clock.cc:648] HybridClock initialized: now 1781992534351504 us; error 31 us; skew 500 ppm
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:34.354367 21229 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:34.355357 21229 webserver.cc:460] Webserver started at http://127.17.67.254:38233/ using document root <none> and password file <none>
I20260620 21:55:34.355902 21229 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:34.355991 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:34.356249 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:34.358352 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "cb2e616c548f44aca1a62ffce5bc9365"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "541edd9e6120e21e07437473f5356b95"
server_key_iv: "fae1f79e033221091baeb9d7f2ab467b"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.358918 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "cb2e616c548f44aca1a62ffce5bc9365"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "541edd9e6120e21e07437473f5356b95"
server_key_iv: "fae1f79e033221091baeb9d7f2ab467b"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.362634 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:55:34.365137 21244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.366590 21229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:34.366693 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "cb2e616c548f44aca1a62ffce5bc9365"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "541edd9e6120e21e07437473f5356b95"
server_key_iv: "fae1f79e033221091baeb9d7f2ab467b"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.366842 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:34.395437 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:34.399166 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:34.399397 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:34.407538 21229 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:40841
I20260620 21:55:34.407555 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:40841 every 8 connection(s)
I20260620 21:55:34.408557 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:34.411561 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:34.414287 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21229
I20260620 21:55:34.414415 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:34.414731 17679 external_mini_cluster.cc:1428] Setting key 7e34f7b44b0ac8342d695e59df1f41bf
I20260620 21:55:34.417850 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365: Bootstrap starting.
I20260620 21:55:34.420048 21297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:34.420780 21297 log.cc:824] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:34.422691 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365: No bootstrap required, opened a new log
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:34.426353 21297 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } }
I20260620 21:55:34.426577 21297 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:34.426636 21297 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2e616c548f44aca1a62ffce5bc9365, State: Initialized, Role: FOLLOWER
I20260620 21:55:34.427138 21297 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } }
I20260620 21:55:34.427297 21297 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:34.427382 21297 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:34.427506 21297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:34.428707 21297 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } }
I20260620 21:55:34.429072 21297 leader_election.cc:302] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb2e616c548f44aca1a62ffce5bc9365; no voters:
I20260620 21:55:34.429391 21297 leader_election.cc:288] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:34.429519 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:34.429807 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [term 1 LEADER]: Becoming Leader. State: Replica: cb2e616c548f44aca1a62ffce5bc9365, State: Running, Role: LEADER
I20260620 21:55:34.429977 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:34.416252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.17.67.254:40841)
0620 21:55:34.416703 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:34.416718 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:34.416755 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:34.417386 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:34.417404 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:34.417834 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:34.418217 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:34.419046 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.420035 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.420912 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.421388 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.421561 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:34.424313 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:34.424319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:34.424332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:34.424360 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:34.426172 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.426678 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.426703 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.426710 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.426837 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.427180 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.427182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.427183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.427771 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:34.427951 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:34.428338 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 21:55:34.428716 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260620 21:55:34.430207 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } }
I20260620 21:55:34.430755 21297 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:34.431469 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb2e616c548f44aca1a62ffce5bc9365. Latest consensus state: current_term: 1 leader_uuid: "cb2e616c548f44aca1a62ffce5bc9365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } } }
I20260620 21:55:34.431639 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:34.432837 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb2e616c548f44aca1a62ffce5bc9365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb2e616c548f44aca1a62ffce5bc9365" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40841 } } }
I20260620 21:55:34.433272 21311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:34.433928 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:34.435989 21311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:34.441527 21311 catalog_manager.cc:1347] Generated new cluster ID: 31f84f8534f84bfd975daa20c669892f
I20260620 21:55:34.441625 21311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:34.447414 21311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:34.448351 21311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:34.462733 21311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb2e616c548f44aca1a62ffce5bc9365: Generated new TSK 0
I20260620 21:55:34.463594 21311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:34.496668 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:34.632393 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:34.632699 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:34.632742 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:34.636961 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:34.637022 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:34.637079 21323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:34.642174 21323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:34.643141 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:34.644182 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:34.653822 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.653841 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.653822 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:34.654247 21323 server_base.cc:1034] running on GCE node
I20260620 21:55:34.654654 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:34.655272 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:34.656445 21323 hybrid_clock.cc:648] HybridClock initialized: now 1781992534656427 us; error 29 us; skew 500 ppm
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:34.659186 21323 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:34.660022 21323 webserver.cc:460] Webserver started at http://127.17.67.193:40021/ using document root <none> and password file <none>
I20260620 21:55:34.660574 21323 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:34.660665 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:34.660907 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:34.662977 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "375a9868da644c199344f0e9fe83704d"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "6f7e2462e6b4b3f5e2926df3d38f6f45"
server_key_iv: "4c1a758a08d9f9db5c8c705c059987d8"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.663540 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "375a9868da644c199344f0e9fe83704d"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "6f7e2462e6b4b3f5e2926df3d38f6f45"
server_key_iv: "4c1a758a08d9f9db5c8c705c059987d8"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.667336 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:55:34.669902 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.671289 21323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:55:34.671428 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "375a9868da644c199344f0e9fe83704d"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "6f7e2462e6b4b3f5e2926df3d38f6f45"
server_key_iv: "4c1a758a08d9f9db5c8c705c059987d8"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.671538 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:34.685307 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:34.688329 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:34.688545 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:34.689175 21323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:34.690085 21323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:34.690158 21323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.690230 21323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:34.690274 21323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.699530 21323 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:39623
I20260620 21:55:34.699550 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:39623 every 8 connection(s)
I20260620 21:55:34.700471 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:34.704504 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21323
I20260620 21:55:34.704638 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:34.704967 17679 external_mini_cluster.cc:1428] Setting key 45540e48cc9e99dfc8b847d9f9a5456f
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:34.716511 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:34.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:52513 (local address 127.17.67.254:40841)
0620 21:55:34.702537 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:34.702540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:34.703337 (+ 797us) server_negotiation.cc:407] Connection header received
0620 21:55:34.704913 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:34.704916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:34.704966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:34.705090 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:34.706599 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.707151 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.708684 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.708908 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.709040 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:34.712284 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:34.712288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:34.712291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:34.712318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:34.714213 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.714875 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.714878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.714879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.715037 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.715429 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.715432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.715434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.715636 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:34.715721 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:34.716256 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 21:55:34.716375 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:55:34.717504 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:34.702744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40841 (local address 127.17.67.193:52513)
0620 21:55:34.703175 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:34.703216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:34.704655 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:34.705294 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:34.705304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:34.705832 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:34.706409 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:34.706424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.707287 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.707293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:34.708550 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:34.708559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.709090 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.709094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:34.709321 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:34.710036 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:34.710061 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:34.712060 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:34.714346 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:34.714356 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:34.714378 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:34.714745 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:34.715148 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:34.715153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:34.715158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:34.715327 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:34.715731 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:34.715761 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:34.716033 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 21:55:34.716197 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:55:34.716461 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260620 21:55:34.718901 21452 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40841
I20260620 21:55:34.719192 21452 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:34.719826 21452 heartbeater.cc:511] Master 127.17.67.254:40841 requested a full tablet report, sending...
I20260620 21:55:34.721657 21261 ts_manager.cc:194] Registered new tserver with Master: 375a9868da644c199344f0e9fe83704d (127.17.67.193:39623)
I20260620 21:55:34.723419 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:52513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:34.733201 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:34.845788 21457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:34.846071 21457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:34.846119 21457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:34.850392 21457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:34.850449 21457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:34.850538 21457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:34.855782 21457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:34.856896 21457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:34.858006 21457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:55:34.868079 21457 server_base.cc:1034] running on GCE node
W20260620 21:55:34.868060 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.868038 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:34.868039 21465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:34.868657 21457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:34.869338 21457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:34.870579 21457 hybrid_clock.cc:648] HybridClock initialized: now 1781992534870534 us; error 76 us; skew 500 ppm
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:34.873395 21457 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:34.874428 21457 webserver.cc:460] Webserver started at http://127.17.67.194:46881/ using document root <none> and password file <none>
I20260620 21:55:34.875048 21457 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:34.875137 21457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:34.875394 21457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:34.877543 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "bf5642786ffb49618d47432d7c0da334"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "fc2d8fcd23cb44704502c9b323a6527f"
server_key_iv: "3696fdf28117d527af267884ed69b89f"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.878175 21457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "bf5642786ffb49618d47432d7c0da334"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "fc2d8fcd23cb44704502c9b323a6527f"
server_key_iv: "3696fdf28117d527af267884ed69b89f"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.882033 21457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:34.884465 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.885784 21457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:34.885911 21457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "bf5642786ffb49618d47432d7c0da334"
format_stamp: "Formatted at 2026-06-20 21:55:34 on dist-test-slave-cx8d"
server_key: "fc2d8fcd23cb44704502c9b323a6527f"
server_key_iv: "3696fdf28117d527af267884ed69b89f"
server_key_version: "encryptionkey@0"
I20260620 21:55:34.886022 21457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:34.897409 21457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:34.900547 21457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:34.900758 21457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:34.901394 21457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:34.902361 21457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:34.902434 21457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.902505 21457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:34.902558 21457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:34.911780 21457 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:35105
I20260620 21:55:34.911815 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:35105 every 8 connection(s)
I20260620 21:55:34.912693 21457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:34.919942 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21457
I20260620 21:55:34.920058 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:34.920364 17679 external_mini_cluster.cc:1428] Setting key d607a5e709e16e5a6f28e399098c7855
Jun 20 21:55:34 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:34.927883 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:34.914755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:33347 (local address 127.17.67.254:40841)
0620 21:55:34.914918 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:34.914923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:34.915802 (+ 879us) server_negotiation.cc:407] Connection header received
0620 21:55:34.917248 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:34.917252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:34.917309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:34.917428 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:34.918713 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.919269 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.920612 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.920839 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.920954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:34.923854 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:34.923857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:34.923859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:34.923882 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:34.925357 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.926109 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.926112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.926113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.926190 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:34.926626 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:34.926629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:34.926630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:34.926817 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:34.926896 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:34.927654 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 21:55:34.927761 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:55:34.929171 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:34.915145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40841 (local address 127.17.67.194:33347)
0620 21:55:34.915625 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:34.915668 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:34.917023 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:34.917692 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:34.917699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:34.918073 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:34.918552 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:34.918565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.919400 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.919403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:34.920481 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:34.920491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:34.921191 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:34.921194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:34.921336 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:34.922110 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:34.922127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:34.923666 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:34.925526 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:34.925535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:34.925558 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:34.926004 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:34.926290 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:34.926295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:34.926298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:34.926509 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:34.926945 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:34.926955 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:34.927334 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 21:55:34.927584 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 21:55:34.927985 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:34.930498 21586 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40841
I20260620 21:55:34.930773 21586 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:34.931483 21586 heartbeater.cc:511] Master 127.17.67.254:40841 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:55:34.932662 21261 ts_manager.cc:194] Registered new tserver with Master: bf5642786ffb49618d47432d7c0da334 (127.17.67.194:35105)
I20260620 21:55:34.933367 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:33347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:34.946816 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:35.068578 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:35.068856 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:35.068899 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:35.073230 21591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:35.073299 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:35.073360 21591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:35.078884 21591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:35.079929 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:35.080919 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:35.090858 21596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:35.090845 21597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:35.090881 21599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:35.091259 21591 server_base.cc:1034] running on GCE node
I20260620 21:55:35.091679 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:35.092301 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:35.093523 21591 hybrid_clock.cc:648] HybridClock initialized: now 1781992535093488 us; error 49 us; skew 500 ppm
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992535, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:35.096444 21591 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:35.097366 21591 webserver.cc:460] Webserver started at http://127.17.67.195:37225/ using document root <none> and password file <none>
I20260620 21:55:35.098002 21591 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:35.098095 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:35.098325 21591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:35.100724 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "9c04d0dd73594f66b26e16899da0622d"
format_stamp: "Formatted at 2026-06-20 21:55:35 on dist-test-slave-cx8d"
server_key: "c52d955f56a07328fa6fb160f44a12d6"
server_key_iv: "ad08fbac1e8b8f8a983f2f6fcb480d77"
server_key_version: "encryptionkey@0"
I20260620 21:55:35.101471 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "9c04d0dd73594f66b26e16899da0622d"
format_stamp: "Formatted at 2026-06-20 21:55:35 on dist-test-slave-cx8d"
server_key: "c52d955f56a07328fa6fb160f44a12d6"
server_key_iv: "ad08fbac1e8b8f8a983f2f6fcb480d77"
server_key_version: "encryptionkey@0"
I20260620 21:55:35.105454 21591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:35.107965 21606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:35.109220 21591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:35.109356 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "9c04d0dd73594f66b26e16899da0622d"
format_stamp: "Formatted at 2026-06-20 21:55:35 on dist-test-slave-cx8d"
server_key: "c52d955f56a07328fa6fb160f44a12d6"
server_key_iv: "ad08fbac1e8b8f8a983f2f6fcb480d77"
server_key_version: "encryptionkey@0"
I20260620 21:55:35.109483 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:35.125734 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:35.128675 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:35.128880 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:35.129482 21591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:35.130348 21591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:35.130419 21591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:35.130518 21591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:35.130569 21591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:35.140097 21591 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:37195
I20260620 21:55:35.140116 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:37195 every 8 connection(s)
I20260620 21:55:35.141002 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:35.143741 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21591
I20260620 21:55:35.143889 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:35.144244 17679 external_mini_cluster.cc:1428] Setting key ef07bf757c8a5902d0459b4ade6038fc
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992535, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:35.154333 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:35.142856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48223 (local address 127.17.67.254:40841)
0620 21:55:35.143015 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:35.143018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:35.143926 (+ 908us) server_negotiation.cc:407] Connection header received
0620 21:55:35.145764 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:35.145767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:35.145817 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:35.145895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:35.147064 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.147577 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.148673 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.148884 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.148967 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:35.151272 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:35.151275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:35.151277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:35.151298 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:35.152632 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.153070 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.153072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.153073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.153142 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.153413 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.153414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.153416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.153613 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:35.153710 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:35.154123 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 21:55:35.154218 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:35.155148 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:35.143270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40841 (local address 127.17.67.195:48223)
0620 21:55:35.143766 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:35.143822 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:35.145531 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:35.146049 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:35.146056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:35.146479 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:35.146936 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:35.146949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.147678 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.147680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:35.148551 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:35.148559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.149006 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.149009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:35.149145 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:35.149768 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:35.149783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:35.151133 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:35.152730 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:35.152734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:35.152746 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:35.152990 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:35.153217 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:35.153219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:35.153220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:35.153341 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:35.153710 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:35.153715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:35.153924 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 21:55:35.154077 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 21:55:35.154304 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":100,"threads_started":1}
I20260620 21:55:35.156177 21720 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40841
I20260620 21:55:35.156421 21720 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:35.157011 21720 heartbeater.cc:511] Master 127.17.67.254:40841 requested a full tablet report, sending...
I20260620 21:55:35.158142 21261 ts_manager.cc:194] Registered new tserver with Master: 9c04d0dd73594f66b26e16899da0622d (127.17.67.195:37195)
I20260620 21:55:35.158692 21261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:48223
I20260620 21:55:35.169795 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:35.195642 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:35.186211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.17.67.193:39623)
0620 21:55:35.186504 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:35.186517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:35.186541 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:55:35.186610 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:35.186616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:35.186805 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:35.186926 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:35.187628 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.188404 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.189299 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.189750 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.189833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:35.192354 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:35.192364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:35.192377 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:35.192398 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:35.194009 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.194412 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.194416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.194418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.194489 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.194714 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.194716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.194717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.194927 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:35.195111 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:35.195290 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 21:55:35.195436 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:35.206645 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:35.198606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.17.67.254:40841)
0620 21:55:35.198742 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:35.198748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:35.198844 (+ 96us) server_negotiation.cc:407] Connection header received
0620 21:55:35.199008 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:35.199011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:35.199057 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:35.199152 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:35.199721 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.200234 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.201019 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:35.201248 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:35.201355 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:35.203614 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:35.203618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:35.203621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:35.203646 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:35.204936 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.205331 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.205336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.205338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.205420 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:35.205765 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:35.205768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:35.205770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:35.205983 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:35.206126 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:35.206357 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:55:35.206479 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 21:55:35.207043 21261 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49644
I20260620 21:55:35.207701 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21323
I20260620 21:55:35.213414 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21457
I20260620 21:55:35.218653 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21591
I20260620 21:55:35.223349 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21229
2026-06-20T21:55:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2183 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21732](info): set up 2 sockets
Jun 20 21:55:35 dist-test-slave-cx8d krb5kdc[21732](info): commencing operation
krb5kdc: starting...
W20260620 21:55:36.305721 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:36 dist-test-slave-cx8d krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:36.430631 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44911
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:36.539269 21745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:36.539582 21745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:36.539664 21745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:36.543767 21745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:36.543857 21745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:36.543907 21745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:36.543964 21745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:36.544003 21745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:36.549239 21745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:36.550323 21745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:36.551402 21745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:36.559975 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:36.559998 21751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:36.559980 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:36.560297 21745 server_base.cc:1034] running on GCE node
I20260620 21:55:36.560804 21745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:36.561973 21745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:36.563163 21745 hybrid_clock.cc:648] HybridClock initialized: now 1781992536563142 us; error 31 us; skew 500 ppm
Jun 20 21:55:36 dist-test-slave-cx8d krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:36.570562 21745 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:36.571348 21745 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:36.571420 21745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:36.571606 21745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:36.573853 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "2500bfeb59e647939c95592717167e28"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "919d5de4046c64d12fd6b68897cec8a3"
server_key_iv: "bdc0196b40986c46e61f145d852dc9b9"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.574385 21745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "2500bfeb59e647939c95592717167e28"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "919d5de4046c64d12fd6b68897cec8a3"
server_key_iv: "bdc0196b40986c46e61f145d852dc9b9"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.578706 21745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:36.582315 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:36.584123 21745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:36.584285 21745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "2500bfeb59e647939c95592717167e28"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "919d5de4046c64d12fd6b68897cec8a3"
server_key_iv: "bdc0196b40986c46e61f145d852dc9b9"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.584429 21745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:36.597779 21745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:36.602039 21745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:36.602244 21745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:36.611537 21745 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44911
I20260620 21:55:36.611802 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44911 every 8 connection(s)
I20260620 21:55:36.612358 21745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:36.615433 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:36.616811 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21745
I20260620 21:55:36.616950 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:36.617270 17679 external_mini_cluster.cc:1428] Setting key bbb777ce2e464efb05fc9ca2bde4e289
I20260620 21:55:36.622082 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: Bootstrap starting.
I20260620 21:55:36.624912 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:36.625823 21812 log.cc:824] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:36.628052 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: No bootstrap required, opened a new log
Jun 20 21:55:36 dist-test-slave-cx8d krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:36.631347 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } }
I20260620 21:55:36.631567 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:36.631600 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2500bfeb59e647939c95592717167e28, State: Initialized, Role: FOLLOWER
I20260620 21:55:36.632046 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } }
I20260620 21:55:36.632166 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:36.632206 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:36.632268 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:36.633121 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } }
I20260620 21:55:36.633447 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2500bfeb59e647939c95592717167e28; no voters:
I20260620 21:55:36.633773 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:36.633858 21817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:36.634063 21817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [term 1 LEADER]: Becoming Leader. State: Replica: 2500bfeb59e647939c95592717167e28, State: Running, Role: LEADER
I20260620 21:55:36.634337 21817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } }
I20260620 21:55:36.635059 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:36.635157 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:36.618816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.17.67.254:44911)
0620 21:55:36.619287 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:36.619311 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:36.619363 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:55:36.620434 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:36.620472 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:36.620937 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:36.621351 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:36.622616 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.624372 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.625591 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.626481 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.626691 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:36.629398 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:36.629403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:36.629415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:36.629440 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:36.632024 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:36.632469 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:36.632475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:36.632483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:36.632615 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:36.632902 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:36.632906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:36.632908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:36.633367 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:36.633787 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:36.633970 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 21:55:36.634308 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260620 21:55:36.635490 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2500bfeb59e647939c95592717167e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } } }
I20260620 21:55:36.635601 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:36.635798 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2500bfeb59e647939c95592717167e28. Latest consensus state: current_term: 1 leader_uuid: "2500bfeb59e647939c95592717167e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2500bfeb59e647939c95592717167e28" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44911 } } }
I20260620 21:55:36.635933 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:36.636041 21821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:36.639150 21821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:36.644804 21821 catalog_manager.cc:1347] Generated new cluster ID: f2f953cd21314f35aa634da6918bdf53
I20260620 21:55:36.644870 21821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:36.659960 21821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:36.661229 21821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:36.670816 21821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: Generated new TSK 0
I20260620 21:55:36.671653 21821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:36.703841 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:36.817453 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:36.817831 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:36.817914 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:36.822036 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:36.822122 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:36.822217 21838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:36.827559 21838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:36.828686 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:36.829818 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:36.839515 21844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:36.839511 21846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:36.839510 21843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:36.839759 21838 server_base.cc:1034] running on GCE node
I20260620 21:55:36.840288 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:36.840898 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:36.842095 21838 hybrid_clock.cc:648] HybridClock initialized: now 1781992536842077 us; error 28 us; skew 500 ppm
Jun 20 21:55:36 dist-test-slave-cx8d krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:36.845009 21838 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:36.845820 21838 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:36.845887 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:36.846048 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:36.848001 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "b5a2fd8469f24d338e9616b14a99b9d0"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "af4f144bd767afb7aa02fef40ee8f38a"
server_key_iv: "c6785aaa91217a7e552a0e5ab06b8c73"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.848498 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "b5a2fd8469f24d338e9616b14a99b9d0"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "af4f144bd767afb7aa02fef40ee8f38a"
server_key_iv: "c6785aaa91217a7e552a0e5ab06b8c73"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.852197 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:55:36.854766 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:36.856001 21838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:36.856144 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "b5a2fd8469f24d338e9616b14a99b9d0"
format_stamp: "Formatted at 2026-06-20 21:55:36 on dist-test-slave-cx8d"
server_key: "af4f144bd767afb7aa02fef40ee8f38a"
server_key_iv: "c6785aaa91217a7e552a0e5ab06b8c73"
server_key_version: "encryptionkey@0"
I20260620 21:55:36.856248 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:36.876508 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:36.879820 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:36.880041 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:36.880697 21838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:36.881536 21838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:36.881649 21838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:36.881714 21838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:36.881745 21838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:36.891249 21838 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:33237
I20260620 21:55:36.891266 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:33237 every 8 connection(s)
I20260620 21:55:36.891845 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:36.900988 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21838
I20260620 21:55:36.901129 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
Jun 20 21:55:36 dist-test-slave-cx8d krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:36.901473 17679 external_mini_cluster.cc:1428] Setting key 85653e61fd4d859d8028d4de24c2d9a0
I20260620 21:55:36.906145 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:36.893545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:48639 (local address 127.17.67.254:44911)
0620 21:55:36.893711 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:36.893714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:36.894569 (+ 855us) server_negotiation.cc:407] Connection header received
0620 21:55:36.895854 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:36.895857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:36.895913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:36.896016 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:36.897155 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.897754 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.898929 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.899175 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.899291 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:36.902020 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:36.902023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:36.902025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:36.902049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:36.903537 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:36.904262 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:36.904265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:36.904266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:36.904348 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:36.904794 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:36.904799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:36.904803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:36.905060 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:36.905171 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:36.905741 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 21:55:36.905901 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 21:55:36.906951 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:36.893969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44911 (local address 127.17.67.193:48639)
0620 21:55:36.894418 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:36.894460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:36.895643 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:36.896180 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:36.896187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:36.896542 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:36.897008 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:36.897021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.897893 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.897897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:36.898771 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:36.898779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:36.899329 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:36.899333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:36.899515 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:36.900158 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:36.900175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:36.901800 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:36.903665 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:36.903675 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:36.903697 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:36.904140 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:36.904446 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:36.904452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:36.904456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:36.904680 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:36.905174 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:36.905179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:36.905438 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 21:55:36.905683 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 21:55:36.906004 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260620 21:55:36.908433 21966 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44911
I20260620 21:55:36.908702 21966 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:36.909188 21966 heartbeater.cc:511] Master 127.17.67.254:44911 requested a full tablet report, sending...
I20260620 21:55:36.910791 21776 ts_manager.cc:194] Registered new tserver with Master: b5a2fd8469f24d338e9616b14a99b9d0 (127.17.67.193:33237)
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:36.912020 21776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:48639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:36.928365 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:37.041836 21971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:37.042093 21971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:37.042136 21971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:37.046165 21971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:37.046214 21971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:37.046271 21971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:37.051347 21971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:37.052255 21971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:37.053179 21971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:37.062682 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.062682 21979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.062686 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:37.063103 21971 server_base.cc:1034] running on GCE node
I20260620 21:55:37.063529 21971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:37.064188 21971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:37.065369 21971 hybrid_clock.cc:648] HybridClock initialized: now 1781992537065361 us; error 31 us; skew 500 ppm
Jun 20 21:55:37 dist-test-slave-cx8d krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:37.068035 21971 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:37.068768 21971 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:37.068830 21971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:37.069032 21971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:37.071157 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "08ba9b6a99724ff9a066c8ec8fad4365"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "657c6eb1ce5d93971f472266f95ccad7"
server_key_iv: "e0b2b0833843c9c60558e1186d4d6050"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.071684 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "08ba9b6a99724ff9a066c8ec8fad4365"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "657c6eb1ce5d93971f472266f95ccad7"
server_key_iv: "e0b2b0833843c9c60558e1186d4d6050"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.075349 21971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:55:37.077646 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.078830 21971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:37.078953 21971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "08ba9b6a99724ff9a066c8ec8fad4365"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "657c6eb1ce5d93971f472266f95ccad7"
server_key_iv: "e0b2b0833843c9c60558e1186d4d6050"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.079056 21971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:37.100225 21971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:37.103381 21971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:37.103590 21971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:37.104211 21971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:37.105029 21971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:37.105077 21971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.105116 21971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:37.105171 21971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.115226 21971 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:33793
I20260620 21:55:37.115255 22098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:33793 every 8 connection(s)
I20260620 21:55:37.115804 21971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
Jun 20 21:55:37 dist-test-slave-cx8d krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:37.125313 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 21971
I20260620 21:55:37.125447 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:37.125783 17679 external_mini_cluster.cc:1428] Setting key 4f56449be477b9bd356d084cd376e0fd
I20260620 21:55:37.130090 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.117555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:36281 (local address 127.17.67.254:44911)
0620 21:55:37.117711 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.117715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.118851 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 21:55:37.119738 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.119742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.119795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.119882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:37.121039 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.121614 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.122749 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.122979 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.123092 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.125888 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:37.125892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:37.125897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:37.125929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:37.127706 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.128283 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.128290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.128294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.128387 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.128810 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.128816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.128818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.129024 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:37.129143 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.129764 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.129909 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:55:37.131170 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.117951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44911 (local address 127.17.67.194:36281)
0620 21:55:37.118707 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.118747 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.119542 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.120089 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.120095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.120441 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:37.120898 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.120910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.121740 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.121748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.122624 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.122633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.123129 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.123134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.123303 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.123927 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:37.123944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:37.125691 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:37.127852 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:37.127859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:37.127876 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:37.128167 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:37.128512 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:37.128516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:37.128518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:37.128703 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:37.129141 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:37.129149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:37.129429 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.129705 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.130081 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260620 21:55:37.132601 22099 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44911
I20260620 21:55:37.132872 22099 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:37.133436 22099 heartbeater.cc:511] Master 127.17.67.254:44911 requested a full tablet report, sending...
I20260620 21:55:37.134760 21776 ts_manager.cc:194] Registered new tserver with Master: 08ba9b6a99724ff9a066c8ec8fad4365 (127.17.67.194:33793)
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:37.135494 21776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:36281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:37.152712 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:37.263494 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:37.263793 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:37.263839 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:37.267910 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:37.267963 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:37.268051 22104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:37.273257 22104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:37.274323 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:37.275370 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:37.285013 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.285017 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.285030 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:37.285432 22104 server_base.cc:1034] running on GCE node
I20260620 21:55:37.285931 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:37.286577 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:37.287760 22104 hybrid_clock.cc:648] HybridClock initialized: now 1781992537287726 us; error 47 us; skew 500 ppm
Jun 20 21:55:37 dist-test-slave-cx8d krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:37.290702 22104 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:37.291479 22104 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:37.291574 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:37.291805 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:37.293953 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "1f73eb76dea440faa193f1d5be9ccf5f"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "3089e6ec6c5210e526c8ee2ba8f8e9a7"
server_key_iv: "e79a614fe2be54b91a7a2fe01bf69732"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.294529 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "1f73eb76dea440faa193f1d5be9ccf5f"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "3089e6ec6c5210e526c8ee2ba8f8e9a7"
server_key_iv: "e79a614fe2be54b91a7a2fe01bf69732"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.298280 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:55:37.300840 22118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.302300 22104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:37.302464 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "1f73eb76dea440faa193f1d5be9ccf5f"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "3089e6ec6c5210e526c8ee2ba8f8e9a7"
server_key_iv: "e79a614fe2be54b91a7a2fe01bf69732"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.302582 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:37.317945 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:37.321375 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:37.321668 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:37.322311 22104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:37.323242 22104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:37.323320 22104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.323393 22104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:37.323442 22104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.333221 22104 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33137
I20260620 21:55:37.333232 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33137 every 8 connection(s)
I20260620 21:55:37.333961 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:37.339408 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22104
I20260620 21:55:37.339543 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:37.339895 17679 external_mini_cluster.cc:1428] Setting key 1aa3ccc646783acf0ce2c40182d2c38d
Jun 20 21:55:37 dist-test-slave-cx8d krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:37.348398 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.335820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:55481 (local address 127.17.67.254:44911)
0620 21:55:37.336069 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.336072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.337297 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 21:55:37.338370 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.338374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.338423 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.338499 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:37.339899 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.340541 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.341737 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.342018 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.342130 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.344835 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:37.344838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:37.344840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:37.344863 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:37.346347 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.346865 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.346877 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.346879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.346965 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.347311 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.347316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.347318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.347506 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:37.347615 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.348110 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.348232 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 21:55:37.349237 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.336342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44911 (local address 127.17.67.195:55481)
0620 21:55:37.337151 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.337195 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.338159 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.338636 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.338644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.339014 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:37.339733 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.339756 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.340668 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.340671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.341618 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.341628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.342185 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.342189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.342363 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.342938 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:37.342956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:37.344663 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:37.346484 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:37.346490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:37.346505 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:37.346764 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:37.347070 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:37.347073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:37.347075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:37.347217 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:37.347621 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:37.347630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:37.347877 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.348060 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.348391 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260620 21:55:37.350448 22232 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44911
I20260620 21:55:37.350687 22232 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:37.351143 22232 heartbeater.cc:511] Master 127.17.67.254:44911 requested a full tablet report, sending...
I20260620 21:55:37.352186 21776 ts_manager.cc:194] Registered new tserver with Master: 1f73eb76dea440faa193f1d5be9ccf5f (127.17.67.195:33137)
I20260620 21:55:37.353053 21776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:55481
I20260620 21:55:37.353928 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:55:37.362974 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.355904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.17.67.254:44911)
0620 21:55:37.356039 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.356048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.356190 (+ 142us) server_negotiation.cc:407] Connection header received
0620 21:55:37.356255 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.356257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.356298 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.356368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:37.357196 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.357777 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.358570 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.358771 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.358847 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.360206 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:37.360209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:37.360211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:37.360229 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:37.361504 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.361959 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.361962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.361964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.362084 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:37.362326 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:37.362328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:37.362329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:37.362526 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:37.362626 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.362784 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.362867 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:55:37.365795 21776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:37.368710 21776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:37.384023 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33137 (local address 127.0.0.1:34254)
0620 21:55:37.378531 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.378542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.378618 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.379246 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.379255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.379277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:37.379430 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.379437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380799 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.380803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.382288 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.382298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383460 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.383565 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383602 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.383725 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.383844 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":266,"spinlock_wait_cycles":48384,"thread_start_us":125,"threads_started":1}
I20260620 21:55:37.384351 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.17.67.194:33793)
0620 21:55:37.378660 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.378672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.378689 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:55:37.378748 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.378752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.378931 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.379082 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:37.379529 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380283 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.382698 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383463 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383568 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383686 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.384094 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.384190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":105,"threads_started":1}
I20260620 21:55:37.384353 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.17.67.193:33237)
0620 21:55:37.378266 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.378275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.378707 (+ 432us) server_negotiation.cc:407] Connection header received
0620 21:55:37.378776 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.378780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.378930 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.379088 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:37.379376 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380621 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.382738 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383342 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383473 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383619 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.384101 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.384209 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":92,"threads_started":1}
I20260620 21:55:37.384608 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.17.67.195:33137)
0620 21:55:37.378417 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.378429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.378632 (+ 203us) server_negotiation.cc:407] Connection header received
0620 21:55:37.378718 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.378725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.378931 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.379082 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:37.379563 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380670 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.382414 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383327 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383444 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383600 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.384227 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.384317 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":155,"threads_started":1}
I20260620 21:55:37.384034 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:33237 (local address 127.0.0.1:38308)
0620 21:55:37.378320 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.378335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.378415 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.379081 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.379087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.379107 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:37.379247 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.379253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380799 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.380803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.382604 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.382614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383703 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.383803 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383826 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.383890 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.383944 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":288,"thread_start_us":103,"threads_started":1}
I20260620 21:55:37.384672 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.377069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33793 (local address 127.0.0.1:45006)
0620 21:55:37.377907 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.377968 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.378096 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.379248 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.379265 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.379283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:37.379416 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.379421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.380421 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.380428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.382556 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.382570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.383750 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.383754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.383847 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.383871 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.383944 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.384018 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":290,"thread_start_us":83,"threads_started":1}
I20260620 21:55:37.387817 22166 tablet_service.cc:1467] Processing CreateTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 (DEFAULT_TABLE table=test-table [id=41865ef7d84443b9bc2a7473dcbe0a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:37.387817 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 (DEFAULT_TABLE table=test-table [id=41865ef7d84443b9bc2a7473dcbe0a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:37.388928 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f9c01938ba4b9197cad6401aa118b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:37.388931 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f9c01938ba4b9197cad6401aa118b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:37.390689 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 (DEFAULT_TABLE table=test-table [id=41865ef7d84443b9bc2a7473dcbe0a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:37.391616 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f9c01938ba4b9197cad6401aa118b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:37.396253 22250 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: Bootstrap starting.
I20260620 21:55:37.396674 22249 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Bootstrap starting.
I20260620 21:55:37.397469 22251 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Bootstrap starting.
I20260620 21:55:37.399448 22251 tablet_bootstrap.cc:654] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:37.399572 22250 tablet_bootstrap.cc:654] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:37.400341 22249 tablet_bootstrap.cc:654] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:37.400484 22251 log.cc:824] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:37.400842 22250 log.cc:824] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:37.401473 22249 log.cc:824] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:37.402683 22251 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: No bootstrap required, opened a new log
I20260620 21:55:37.402884 22251 ts_tablet_manager.cc:1397] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:37.403018 22250 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: No bootstrap required, opened a new log
I20260620 21:55:37.403195 22249 tablet_bootstrap.cc:492] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: No bootstrap required, opened a new log
I20260620 21:55:37.403213 22250 ts_tablet_manager.cc:1397] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:55:37.403374 22249 ts_tablet_manager.cc:1397] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:55:37.406565 22249 raft_consensus.cc:348] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.406809 22249 raft_consensus.cc:374] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:37.406934 22249 raft_consensus.cc:729] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f73eb76dea440faa193f1d5be9ccf5f, State: Initialized, Role: FOLLOWER
I20260620 21:55:37.406960 22251 raft_consensus.cc:348] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.406955 22250 raft_consensus.cc:348] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.407213 22250 raft_consensus.cc:374] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:37.407213 22251 raft_consensus.cc:374] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:37.407270 22251 raft_consensus.cc:729] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ba9b6a99724ff9a066c8ec8fad4365, State: Initialized, Role: FOLLOWER
I20260620 21:55:37.407270 22250 raft_consensus.cc:729] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a2fd8469f24d338e9616b14a99b9d0, State: Initialized, Role: FOLLOWER
I20260620 21:55:37.407454 22249 consensus_queue.cc:260] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.407691 22251 consensus_queue.cc:260] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.407759 22250 consensus_queue.cc:260] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.408186 22232 heartbeater.cc:503] Master 127.17.67.254:44911 was elected leader, sending a full tablet report...
I20260620 21:55:37.408459 22249 ts_tablet_manager.cc:1428] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:55:37.408592 22251 ts_tablet_manager.cc:1428] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:37.408880 22099 heartbeater.cc:503] Master 127.17.67.254:44911 was elected leader, sending a full tablet report...
I20260620 21:55:37.409420 21966 heartbeater.cc:503] Master 127.17.67.254:44911 was elected leader, sending a full tablet report...
I20260620 21:55:37.409523 22250 ts_tablet_manager.cc:1428] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:37.415345 22255 raft_consensus.cc:482] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:37.415465 22255 raft_consensus.cc:504] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.416353 22255 leader_election.cc:288] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ba9b6a99724ff9a066c8ec8fad4365 (127.17.67.194:33793), b5a2fd8469f24d338e9616b14a99b9d0 (127.17.67.193:33237)
I20260620 21:55:37.420262 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.416707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:57199 (local address 127.17.67.193:33237)
0620 21:55:37.416812 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.416818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.417260 (+ 442us) server_negotiation.cc:407] Connection header received
0620 21:55:37.417345 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.417348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.417395 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.417478 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:37.417766 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.418271 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419288 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.419688 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.419863 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.420073 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.420140 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:55:37.420264 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.416559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33793 (local address 127.17.67.195:33401)
0620 21:55:37.416683 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.416700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.416782 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.417309 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.417314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.417336 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:37.417466 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.417475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.418263 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.418266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.419167 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.419176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.419803 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.419971 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.419988 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.420080 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.420145 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":64}
I20260620 21:55:37.420265 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.416939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:33401 (local address 127.17.67.194:33793)
0620 21:55:37.417082 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.417087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.417100 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:37.417142 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.417145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.417199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.417275 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:37.417587 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.418162 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419288 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.419686 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419812 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.419866 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.420105 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.420168 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:55:37.420287 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.416554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:33237 (local address 127.17.67.195:57199)
0620 21:55:37.417147 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.417160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.417255 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.417503 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.417505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.417538 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:37.417664 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.417670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.418376 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.418378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.419170 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:37.419177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.419807 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.419810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:37.419939 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.419975 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.420034 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.420123 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":89,"thread_start_us":99,"threads_started":1}
I20260620 21:55:37.420826 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f9c01938ba4b9197cad6401aa118b9" candidate_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" is_pre_election: true
I20260620 21:55:37.420919 22034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f9c01938ba4b9197cad6401aa118b9" candidate_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" is_pre_election: true
I20260620 21:55:37.421113 21920 raft_consensus.cc:2454] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f73eb76dea440faa193f1d5be9ccf5f in term 0.
I20260620 21:55:37.421226 22034 raft_consensus.cc:2454] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f73eb76dea440faa193f1d5be9ccf5f in term 0.
I20260620 21:55:37.421622 22120 leader_election.cc:302] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f73eb76dea440faa193f1d5be9ccf5f, b5a2fd8469f24d338e9616b14a99b9d0; no voters:
I20260620 21:55:37.421865 22255 raft_consensus.cc:2781] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:37.421943 22255 raft_consensus.cc:482] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:37.421993 22255 raft_consensus.cc:3037] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:37.422904 22255 raft_consensus.cc:504] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.423205 22255 leader_election.cc:288] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [CANDIDATE]: Term 1 election: Requested vote from peers 08ba9b6a99724ff9a066c8ec8fad4365 (127.17.67.194:33793), b5a2fd8469f24d338e9616b14a99b9d0 (127.17.67.193:33237)
I20260620 21:55:37.423558 22034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f9c01938ba4b9197cad6401aa118b9" candidate_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ba9b6a99724ff9a066c8ec8fad4365"
I20260620 21:55:37.423558 21920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f9c01938ba4b9197cad6401aa118b9" candidate_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a2fd8469f24d338e9616b14a99b9d0"
I20260620 21:55:37.423692 22034 raft_consensus.cc:3037] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:37.423692 21920 raft_consensus.cc:3037] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:37.425053 22034 raft_consensus.cc:2454] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f73eb76dea440faa193f1d5be9ccf5f in term 1.
I20260620 21:55:37.425053 21920 raft_consensus.cc:2454] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f73eb76dea440faa193f1d5be9ccf5f in term 1.
I20260620 21:55:37.425446 22120 leader_election.cc:302] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f73eb76dea440faa193f1d5be9ccf5f, b5a2fd8469f24d338e9616b14a99b9d0; no voters:
I20260620 21:55:37.425678 22255 raft_consensus.cc:2781] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:37.425940 22255 raft_consensus.cc:686] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 1 LEADER]: Becoming Leader. State: Replica: 1f73eb76dea440faa193f1d5be9ccf5f, State: Running, Role: LEADER
I20260620 21:55:37.426267 22255 consensus_queue.cc:237] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } }
I20260620 21:55:37.429407 21776 catalog_manager.cc:5495] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f73eb76dea440faa193f1d5be9ccf5f (127.17.67.195). New cstate: current_term: 1 leader_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f73eb76dea440faa193f1d5be9ccf5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33137 } health_report { overall_health: HEALTHY } } }
I20260620 21:55:37.445393 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.441421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.17.67.195:33137)
0620 21:55:37.441615 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.441623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.441641 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:37.441880 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.441883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.441943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.442063 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:37.442345 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.442897 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.443826 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:37.444070 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:37.444186 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:37.444513 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:37.444952 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:37.445098 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.445145 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.445205 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 21:55:37.451545 21920 raft_consensus.cc:1261] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 1 FOLLOWER]: Refusing update from remote peer 1f73eb76dea440faa193f1d5be9ccf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:37.451545 22034 raft_consensus.cc:1261] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 1 FOLLOWER]: Refusing update from remote peer 1f73eb76dea440faa193f1d5be9ccf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:37.452473 22259 consensus_queue.cc:1035] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ba9b6a99724ff9a066c8ec8fad4365" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:37.452673 22255 consensus_queue.cc:1035] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a2fd8469f24d338e9616b14a99b9d0" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:37.469470 21775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:37.469686 21775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:37.472065 21775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2500bfeb59e647939c95592717167e28: Sending DeleteTablet for 3 replicas of tablet 56f9c01938ba4b9197cad6401aa118b9
I20260620 21:55:37.472776 21900 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:37 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:38308
I20260620 21:55:37.472844 22033 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:37 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:45006
I20260620 21:55:37.473266 22272 tablet_replica.cc:330] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0: stopping tablet replica
I20260620 21:55:37.473549 22166 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f9c01938ba4b9197cad6401aa118b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:37 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:34254
I20260620 21:55:37.473748 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21838
I20260620 21:55:37.473744 22272 raft_consensus.cc:2229] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:37.474112 22272 raft_consensus.cc:2258] T 56f9c01938ba4b9197cad6401aa118b9 P b5a2fd8469f24d338e9616b14a99b9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:37.474468 22273 tablet_replica.cc:330] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: stopping tablet replica
I20260620 21:55:37.474763 22273 raft_consensus.cc:2229] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:37.475013 22273 raft_consensus.cc:2258] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:37.476720 22273 ts_tablet_manager.cc:1905] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:37.480154 22273 ts_tablet_manager.cc:1918] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:37.480258 22273 log.cc:1196] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/56f9c01938ba4b9197cad6401aa118b9
I20260620 21:55:37.480535 22274 tablet_replica.cc:330] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: stopping tablet replica
I20260620 21:55:37.480583 22273 ts_tablet_manager.cc:1939] T 56f9c01938ba4b9197cad6401aa118b9 P 08ba9b6a99724ff9a066c8ec8fad4365: Deleting consensus metadata
I20260620 21:55:37.480922 22274 raft_consensus.cc:2229] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:37.481361 22274 raft_consensus.cc:2258] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:55:37.482237 21761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:33237 (error 108)
I20260620 21:55:37.482487 21762 catalog_manager.cc:4841] TS 08ba9b6a99724ff9a066c8ec8fad4365 (127.17.67.194:33793): tablet 56f9c01938ba4b9197cad6401aa118b9 (table test-table [id=41865ef7d84443b9bc2a7473dcbe0a63]) successfully deleted
I20260620 21:55:37.483239 22274 ts_tablet_manager.cc:1905] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:37.483278 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21971
I20260620 21:55:37.483541 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.483277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:33237 (local address 127.0.0.1:38314)
0620 21:55:37.483400 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.483465 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 21:55:37.484746 21761 catalog_manager.cc:4584] TS b5a2fd8469f24d338e9616b14a99b9d0 (127.17.67.193:33237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56f9c01938ba4b9197cad6401aa118b9: Network error: Client connection negotiation failed: client connection to 127.17.67.193:33237: connect: Connection refused (error 111)
I20260620 21:55:37.487533 22274 ts_tablet_manager.cc:1918] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:37.487676 22274 log.cc:1196] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/56f9c01938ba4b9197cad6401aa118b9
I20260620 21:55:37.488049 22274 ts_tablet_manager.cc:1939] T 56f9c01938ba4b9197cad6401aa118b9 P 1f73eb76dea440faa193f1d5be9ccf5f: Deleting consensus metadata
I20260620 21:55:37.488511 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22104
I20260620 21:55:37.493957 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 21745
2026-06-20T21:55:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2270 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T21:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:37Z Disabled control of system clock
I20260620 21:55:37.523878 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41809
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:55:37.631266 22279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:37.631515 22279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:37.631551 22279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:37.635490 22279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:37.635543 22279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:37.635560 22279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:37.635576 22279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:37.635591 22279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:37.640223 22279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:37.641300 22279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:37.642601 22279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:37.650801 22285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:37.650848 22279 server_base.cc:1034] running on GCE node
W20260620 21:55:37.650761 22284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.650794 22287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:37.651476 22279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:37.652504 22279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:37.653630 22279 hybrid_clock.cc:648] HybridClock initialized: now 1781992537653600 us; error 43 us; skew 500 ppm
I20260620 21:55:37.655339 22279 webserver.cc:460] Webserver started at http://127.17.67.254:42135/ using document root <none> and password file <none>
I20260620 21:55:37.655825 22279 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:37.655881 22279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:37.656042 22279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:37.657989 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "32a39ff2c75c466c9a84498ec267dad3"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "b998dab107a67c4595ad12b6273beeaf"
server_key_iv: "77abe58703105502eb3a3644c55f7cdf"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.658474 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "32a39ff2c75c466c9a84498ec267dad3"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "b998dab107a67c4595ad12b6273beeaf"
server_key_iv: "77abe58703105502eb3a3644c55f7cdf"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.661947 22279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:37.664258 22293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.665475 22279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:37.665620 22279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "32a39ff2c75c466c9a84498ec267dad3"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "b998dab107a67c4595ad12b6273beeaf"
server_key_iv: "77abe58703105502eb3a3644c55f7cdf"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.665726 22279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:37.670184 22279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:37.670746 22279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:37.670938 22279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:37.677618 22279 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41809
I20260620 21:55:37.677626 22345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41809 every 8 connection(s)
I20260620 21:55:37.678488 22279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:37.679356 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22279
I20260620 21:55:37.679477 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:37.679761 17679 external_mini_cluster.cc:1428] Setting key 93b2f09b2d8c566fbf87389c0d11c485
I20260620 21:55:37.681437 22346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:37.686149 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.681125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.17.67.254:41809)
0620 21:55:37.682264 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.682283 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.682325 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:55:37.683172 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.683194 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.683633 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.684007 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:37.684690 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:37.684695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:37.684709 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:37.684719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:37.684791 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:37.684813 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:37.684961 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.685033 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.685265 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":626,"threads_started":1}
I20260620 21:55:37.688037 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: Bootstrap starting.
I20260620 21:55:37.690105 22346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:37.690805 22346 log.cc:824] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:37.693612 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: No bootstrap required, opened a new log
I20260620 21:55:37.697311 22346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } }
I20260620 21:55:37.697536 22346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:37.697638 22346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a39ff2c75c466c9a84498ec267dad3, State: Initialized, Role: FOLLOWER
I20260620 21:55:37.698185 22346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } }
I20260620 21:55:37.698369 22346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:37.698439 22346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:37.698541 22346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:37.699496 22346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } }
I20260620 21:55:37.699868 22346 leader_election.cc:302] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a39ff2c75c466c9a84498ec267dad3; no voters:
I20260620 21:55:37.700204 22346 leader_election.cc:288] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:37.700338 22351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:37.700585 22351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [term 1 LEADER]: Becoming Leader. State: Replica: 32a39ff2c75c466c9a84498ec267dad3, State: Running, Role: LEADER
I20260620 21:55:37.700878 22351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } }
I20260620 21:55:37.701493 22346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:37.702587 22352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a39ff2c75c466c9a84498ec267dad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } } }
I20260620 21:55:37.702587 22353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a39ff2c75c466c9a84498ec267dad3. Latest consensus state: current_term: 1 leader_uuid: "32a39ff2c75c466c9a84498ec267dad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a39ff2c75c466c9a84498ec267dad3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41809 } } }
I20260620 21:55:37.702732 22352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:37.702732 22353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:37.703187 22360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:37.706275 22360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:37.711432 22360 catalog_manager.cc:1347] Generated new cluster ID: 3c60d16b1b1f46a8bdca3236c7d35178
I20260620 21:55:37.711515 22360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:37.728256 22360 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:37.728595 22360 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:37.739485 22360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: Generated new TSK 0
I20260620 21:55:37.740233 22360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:37.751542 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--builtin_ntp_servers=127.17.67.212:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:55:37.862679 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:37.863025 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:37.863070 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:37.867252 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:37.867300 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:37.867411 22370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:37.874972 22370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:37.876063 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:37.877606 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:37.887655 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.887600 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:37.887593 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:37.889488 22370 server_base.cc:1034] running on GCE node
I20260620 21:55:37.889966 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:37.890643 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:37.891824 22370 hybrid_clock.cc:648] HybridClock initialized: now 1781992537891805 us; error 32 us; skew 500 ppm
I20260620 21:55:37.893930 22370 webserver.cc:460] Webserver started at http://127.17.67.193:41637/ using document root <none> and password file <none>
I20260620 21:55:37.894506 22370 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:37.894569 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:37.894742 22370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:37.896895 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "96ed285608df4f97abd85f7554e4cf3c"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "7d83e0169981f72d5a4b691c8a72cc02"
server_key_iv: "f7e36e94d2a6bb11f050d4ff946af64b"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.897531 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "96ed285608df4f97abd85f7554e4cf3c"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "7d83e0169981f72d5a4b691c8a72cc02"
server_key_iv: "f7e36e94d2a6bb11f050d4ff946af64b"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.901434 22370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:37.904356 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.905722 22370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:37.905881 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "96ed285608df4f97abd85f7554e4cf3c"
format_stamp: "Formatted at 2026-06-20 21:55:37 on dist-test-slave-cx8d"
server_key: "7d83e0169981f72d5a4b691c8a72cc02"
server_key_iv: "f7e36e94d2a6bb11f050d4ff946af64b"
server_key_version: "encryptionkey@0"
I20260620 21:55:37.905990 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:37.910699 22370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:37.911321 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:37.911504 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:37.912045 22370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:37.912935 22370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:37.912981 22370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.913044 22370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:37.913076 22370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:37.921672 22370 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:40689
I20260620 21:55:37.921692 22497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:40689 every 8 connection(s)
I20260620 21:55:37.922561 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:37.927893 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22370
I20260620 21:55:37.928041 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:37.928359 17679 external_mini_cluster.cc:1428] Setting key 57a9ca3cb3abdd0770614336a058e628
I20260620 21:55:37.929513 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.924589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:43033 (local address 127.17.67.254:41809)
0620 21:55:37.924749 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:37.924754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:37.925994 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 21:55:37.927453 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:37.927456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:37.927506 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:37.927594 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:37.928984 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:37.928989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:37.928992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:37.929007 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:37.929035 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:37.929045 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:37.929160 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:37.929323 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 21:55:37.929368 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:55:37.930466 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:37.925043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41809 (local address 127.17.67.193:43033)
0620 21:55:37.925798 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:37.925842 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:37.927202 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:37.927764 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:37.927771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:37.927864 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:37.928612 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:37.928614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:37.928669 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:37.928761 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:37.929156 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:37.929166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:37.929174 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:55:37.929355 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:55:37.929617 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260620 21:55:37.930480 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--builtin_ntp_servers=127.17.67.212:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 21:55:37.932049 22498 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41809
I20260620 21:55:37.932293 22498 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:37.932691 22498 heartbeater.cc:511] Master 127.17.67.254:41809 requested a full tablet report, sending...
I20260620 21:55:37.934322 22310 ts_manager.cc:194] Registered new tserver with Master: 96ed285608df4f97abd85f7554e4cf3c (127.17.67.193:40689)
W20260620 21:55:38.042485 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:38.042783 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:38.042857 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:38.046942 22501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:38.047015 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:38.047103 22501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:38.051844 22501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:38.052933 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:38.054332 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:38.063796 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:38.064015 22501 server_base.cc:1034] running on GCE node
W20260620 21:55:38.063788 22506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:38.063788 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:38.064761 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:38.065411 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:38.066636 22501 hybrid_clock.cc:648] HybridClock initialized: now 1781992538066593 us; error 58 us; skew 500 ppm
I20260620 21:55:38.068691 22501 webserver.cc:460] Webserver started at http://127.17.67.194:34429/ using document root <none> and password file <none>
I20260620 21:55:38.069365 22501 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:38.069460 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:38.069746 22501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:38.071982 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "52d5abd2251e4b1dbdd3e60308ccd315"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "667624891565238d3a518d7512ee9bb2"
server_key_iv: "756b869aacced7480b51a0e10bea34da"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.072729 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "52d5abd2251e4b1dbdd3e60308ccd315"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "667624891565238d3a518d7512ee9bb2"
server_key_iv: "756b869aacced7480b51a0e10bea34da"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.076503 22501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:55:38.079499 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.080799 22501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:38.080925 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "52d5abd2251e4b1dbdd3e60308ccd315"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "667624891565238d3a518d7512ee9bb2"
server_key_iv: "756b869aacced7480b51a0e10bea34da"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.081015 22501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:38.085451 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:38.086071 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:38.086210 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:38.086671 22501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:38.087587 22501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:38.087632 22501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.087669 22501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:38.087684 22501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.096688 22501 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:35209
I20260620 21:55:38.096716 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:35209 every 8 connection(s)
I20260620 21:55:38.097685 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:38.104001 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.099539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:57215 (local address 127.17.67.254:41809)
0620 21:55:38.099708 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.099713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.100759 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 21:55:38.102183 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.102186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.102222 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.102298 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.103466 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.103469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.103471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.103482 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.103500 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.103508 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.103597 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.103823 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.103864 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:55:38.104759 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.099985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41809 (local address 127.17.67.194:57215)
0620 21:55:38.100619 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.100658 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.101993 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.102450 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.102456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.102597 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.103199 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.103200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.103238 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.103302 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.103586 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.103593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.103618 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.103787 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.103999 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260620 21:55:38.105964 22629 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41809
I20260620 21:55:38.106181 22629 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:38.106564 22629 heartbeater.cc:511] Master 127.17.67.254:41809 requested a full tablet report, sending...
I20260620 21:55:38.107563 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22501
I20260620 21:55:38.107581 22310 ts_manager.cc:194] Registered new tserver with Master: 52d5abd2251e4b1dbdd3e60308ccd315 (127.17.67.194:35209)
I20260620 21:55:38.107693 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:38.107998 17679 external_mini_cluster.cc:1428] Setting key 4c5c0ea33f4f09a7107ba75f38c4b198
I20260620 21:55:38.109894 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--builtin_ntp_servers=127.17.67.212:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:55:38.223212 22632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:38.223470 22632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:38.223508 22632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:38.227530 22632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:38.227593 22632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:38.227651 22632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:38.232553 22632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:38.233702 22632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:38.235145 22632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:38.245151 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:38.245154 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:38.245154 22640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:38.245709 22632 server_base.cc:1034] running on GCE node
I20260620 21:55:38.246069 22632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:38.246661 22632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:38.247819 22632 hybrid_clock.cc:648] HybridClock initialized: now 1781992538247782 us; error 52 us; skew 500 ppm
I20260620 21:55:38.249806 22632 webserver.cc:460] Webserver started at http://127.17.67.195:41055/ using document root <none> and password file <none>
I20260620 21:55:38.250392 22632 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:38.250458 22632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:38.250674 22632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:38.252736 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "9bca8fadc0494f4ea25eb0bd1065547e"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "4ab991e4ecd3d9dc898cafe9670f13a2"
server_key_iv: "b7765665b432c05e8145363dc239a0ec"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.253309 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "9bca8fadc0494f4ea25eb0bd1065547e"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "4ab991e4ecd3d9dc898cafe9670f13a2"
server_key_iv: "b7765665b432c05e8145363dc239a0ec"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.257126 22632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:38.259749 22646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.261157 22632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:38.261314 22632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "9bca8fadc0494f4ea25eb0bd1065547e"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "4ab991e4ecd3d9dc898cafe9670f13a2"
server_key_iv: "b7765665b432c05e8145363dc239a0ec"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.261433 22632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:38.266697 22632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:38.267350 22632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:38.267556 22632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:38.268139 22632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:38.269151 22632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:38.269200 22632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.269265 22632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:38.269327 22632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.279137 22632 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:42589
I20260620 21:55:38.279155 22759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:42589 every 8 connection(s)
I20260620 21:55:38.280133 22632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:38.286478 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22632
I20260620 21:55:38.286598 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.282066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:41015 (local address 127.17.67.254:41809)
0620 21:55:38.282217 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.282222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.283397 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 21:55:38.284815 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.284818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.284863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.284949 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.286120 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.286124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.286127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.286140 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.286162 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.286178 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.286290 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.286417 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.286463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:38.286617 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:38.286926 17679 external_mini_cluster.cc:1428] Setting key 6093bbcec6f9f3f6a3a685c34d253988
I20260620 21:55:38.287708 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.282527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41809 (local address 127.17.67.195:41015)
0620 21:55:38.283236 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.283285 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.284571 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.285114 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.285122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.285211 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.285847 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.285848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.285889 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.285963 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.286263 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.286270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.286276 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.286459 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.286788 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260620 21:55:38.288793 22760 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41809
I20260620 21:55:38.289016 22760 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:38.289417 22760 heartbeater.cc:511] Master 127.17.67.254:41809 requested a full tablet report, sending...
I20260620 21:55:38.290441 22309 ts_manager.cc:194] Registered new tserver with Master: 9bca8fadc0494f4ea25eb0bd1065547e (127.17.67.195:42589)
I20260620 21:55:38.300971 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:55:38.304867 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.303140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.17.67.254:41809)
0620 21:55:38.303282 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.303286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.303557 (+ 271us) server_negotiation.cc:407] Connection header received
0620 21:55:38.303645 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.303647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.303680 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.303750 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.304529 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.304532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.304534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.304543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.304561 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.304568 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.304636 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.304698 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.304730 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:55:38.308254 22309 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:38.311515 22309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:38.324173 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.321516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.17.67.195:42589)
0620 21:55:38.322223 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.322234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.322255 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:55:38.322302 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.322306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.322465 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.322592 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.323587 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.323594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.323607 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.323618 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.323665 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.323688 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.323805 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.323948 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.324020 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":175,"threads_started":1}
I20260620 21:55:38.324417 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.320704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.17.67.193:40689)
0620 21:55:38.321929 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.321942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.321964 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:55:38.322017 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.322024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.322214 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.322522 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.323609 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.323615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.323627 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.323639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.323681 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.323728 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.324064 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.324104 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.324165 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":128,"threads_started":1}
I20260620 21:55:38.324545 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.320588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40689 (local address 127.0.0.1:35422)
0620 21:55:38.320942 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.321233 (+ 291us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.321339 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.322364 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.322367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.322376 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.323006 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.323008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.323028 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.323458 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.323876 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.323878 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.323880 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.323938 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.324029 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260620 21:55:38.324769 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.321357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:42589 (local address 127.0.0.1:40532)
0620 21:55:38.321915 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.321927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.321998 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.322930 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.322934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.322949 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.323370 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.323372 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.323380 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.323429 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.323777 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.323786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.323791 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.323933 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.324659 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":81,"threads_started":1}
I20260620 21:55:38.325007 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.320610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:35209 (local address 127.0.0.1:57704)
0620 21:55:38.321599 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.321613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.321690 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.322269 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.322278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.322360 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.323471 (+ 1111us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.323475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.323487 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.323513 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.323871 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.323874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.323878 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.323947 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.324004 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"spinlock_wait_cycles":6400,"thread_start_us":109,"threads_started":1}
I20260620 21:55:38.325655 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.321379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.17.67.194:35209)
0620 21:55:38.321782 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.321795 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.321837 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:55:38.321894 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.321900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.322111 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.322251 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.323640 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.323646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.323654 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.323664 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.323701 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.323726 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.323831 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.324104 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.324166 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":84,"threads_started":1}
I20260620 21:55:38.327723 22694 tablet_service.cc:1467] Processing CreateTablet for tablet 177300352715495a94428caeb9fee55d (DEFAULT_TABLE table=test-table [id=3cff61f8fad9437697d2c80f40e4ad22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:38.327750 22432 tablet_service.cc:1467] Processing CreateTablet for tablet 177300352715495a94428caeb9fee55d (DEFAULT_TABLE table=test-table [id=3cff61f8fad9437697d2c80f40e4ad22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:38.328115 22563 tablet_service.cc:1467] Processing CreateTablet for tablet 177300352715495a94428caeb9fee55d (DEFAULT_TABLE table=test-table [id=3cff61f8fad9437697d2c80f40e4ad22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:38.328765 22694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177300352715495a94428caeb9fee55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:38.328778 22432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177300352715495a94428caeb9fee55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:38.328974 22563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177300352715495a94428caeb9fee55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:38.334676 22777 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Bootstrap starting.
I20260620 21:55:38.335474 22778 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Bootstrap starting.
I20260620 21:55:38.336315 22779 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Bootstrap starting.
I20260620 21:55:38.337359 22777 tablet_bootstrap.cc:654] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:38.337471 22778 tablet_bootstrap.cc:654] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:38.338232 22779 tablet_bootstrap.cc:654] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:38.338378 22778 log.cc:824] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:38.338377 22777 log.cc:824] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:38.339043 22779 log.cc:824] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:38.340374 22777 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: No bootstrap required, opened a new log
I20260620 21:55:38.340376 22778 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: No bootstrap required, opened a new log
I20260620 21:55:38.340564 22778 ts_tablet_manager.cc:1397] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:55:38.340564 22777 ts_tablet_manager.cc:1397] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:38.340708 22779 tablet_bootstrap.cc:492] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: No bootstrap required, opened a new log
I20260620 21:55:38.340916 22779 ts_tablet_manager.cc:1397] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:55:38.344638 22779 raft_consensus.cc:348] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.344893 22779 raft_consensus.cc:374] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:38.344946 22779 raft_consensus.cc:729] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d5abd2251e4b1dbdd3e60308ccd315, State: Initialized, Role: FOLLOWER
I20260620 21:55:38.345048 22777 raft_consensus.cc:348] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.345048 22778 raft_consensus.cc:348] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.345289 22777 raft_consensus.cc:374] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:38.345289 22778 raft_consensus.cc:374] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:38.345345 22777 raft_consensus.cc:729] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bca8fadc0494f4ea25eb0bd1065547e, State: Initialized, Role: FOLLOWER
I20260620 21:55:38.345350 22778 raft_consensus.cc:729] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ed285608df4f97abd85f7554e4cf3c, State: Initialized, Role: FOLLOWER
I20260620 21:55:38.345434 22779 consensus_queue.cc:260] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.345789 22778 consensus_queue.cc:260] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.345789 22777 consensus_queue.cc:260] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.346166 22629 heartbeater.cc:503] Master 127.17.67.254:41809 was elected leader, sending a full tablet report...
I20260620 21:55:38.346779 22777 ts_tablet_manager.cc:1428] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:55:38.346779 22778 ts_tablet_manager.cc:1428] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:55:38.347307 22498 heartbeater.cc:503] Master 127.17.67.254:41809 was elected leader, sending a full tablet report...
I20260620 21:55:38.347466 22779 ts_tablet_manager.cc:1428] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:38.347541 22760 heartbeater.cc:503] Master 127.17.67.254:41809 was elected leader, sending a full tablet report...
W20260620 21:55:38.348613 22630 tablet.cc:2369] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:55:38.423666 22499 tablet.cc:2369] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:55:38.531148 22761 tablet.cc:2369] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:38.631043 22783 raft_consensus.cc:482] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:38.631300 22783 raft_consensus.cc:504] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.632477 22783 leader_election.cc:288] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96ed285608df4f97abd85f7554e4cf3c (127.17.67.193:40689), 9bca8fadc0494f4ea25eb0bd1065547e (127.17.67.195:42589)
I20260620 21:55:38.634685 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.632886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:33931 (local address 127.17.67.193:40689)
0620 21:55:38.633049 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.633054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.633079 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:55:38.633246 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.633253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.633298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.633388 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.634202 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.634206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.634209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.634222 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.634242 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.634253 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.634359 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.634453 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.634509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:55:38.634686 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.632683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40689 (local address 127.17.67.194:33931)
0620 21:55:38.632993 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.633012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.633134 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.633451 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.633453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.633467 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.634023 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.634026 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.634040 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.634075 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.634353 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.634356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.634362 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.634458 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.634499 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260620 21:55:38.634809 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.633214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:38515 (local address 127.17.67.195:42589)
0620 21:55:38.633385 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.633390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.633475 (+ 85us) server_negotiation.cc:407] Connection header received
0620 21:55:38.633556 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.633558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.633633 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.633741 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.634390 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.634396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.634402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.634417 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.634441 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.634451 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.634534 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.634642 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.634679 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:38.634863 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.633078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:42589 (local address 127.17.67.194:38515)
0620 21:55:38.633380 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.633394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:38.633473 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:38.633764 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:38.633767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:38.633796 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:38.634221 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:55:38.634224 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:38.634234 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:55:38.634260 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:38.634540 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:38.634543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:38.634547 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:55:38.634634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:38.634669 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":56,"threads_started":1}
I20260620 21:55:38.635452 22452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177300352715495a94428caeb9fee55d" candidate_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ed285608df4f97abd85f7554e4cf3c" is_pre_election: true
I20260620 21:55:38.635461 22714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177300352715495a94428caeb9fee55d" candidate_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" is_pre_election: true
I20260620 21:55:38.635744 22452 raft_consensus.cc:2454] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d5abd2251e4b1dbdd3e60308ccd315 in term 0.
I20260620 21:55:38.635742 22714 raft_consensus.cc:2454] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52d5abd2251e4b1dbdd3e60308ccd315 in term 0.
I20260620 21:55:38.636183 22517 leader_election.cc:302] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d5abd2251e4b1dbdd3e60308ccd315, 9bca8fadc0494f4ea25eb0bd1065547e; no voters:
I20260620 21:55:38.636477 22783 raft_consensus.cc:2781] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:38.636550 22783 raft_consensus.cc:482] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:38.636580 22783 raft_consensus.cc:3037] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:38.637678 22783 raft_consensus.cc:504] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.638015 22783 leader_election.cc:288] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [CANDIDATE]: Term 1 election: Requested vote from peers 96ed285608df4f97abd85f7554e4cf3c (127.17.67.193:40689), 9bca8fadc0494f4ea25eb0bd1065547e (127.17.67.195:42589)
I20260620 21:55:38.638433 22452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177300352715495a94428caeb9fee55d" candidate_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ed285608df4f97abd85f7554e4cf3c"
I20260620 21:55:38.638520 22714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177300352715495a94428caeb9fee55d" candidate_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bca8fadc0494f4ea25eb0bd1065547e"
I20260620 21:55:38.638535 22452 raft_consensus.cc:3037] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:38.638654 22714 raft_consensus.cc:3037] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:38.639560 22452 raft_consensus.cc:2454] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d5abd2251e4b1dbdd3e60308ccd315 in term 1.
I20260620 21:55:38.639715 22714 raft_consensus.cc:2454] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52d5abd2251e4b1dbdd3e60308ccd315 in term 1.
I20260620 21:55:38.639820 22517 leader_election.cc:302] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52d5abd2251e4b1dbdd3e60308ccd315, 96ed285608df4f97abd85f7554e4cf3c; no voters:
I20260620 21:55:38.639995 22783 raft_consensus.cc:2781] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:38.640223 22783 raft_consensus.cc:686] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 1 LEADER]: Becoming Leader. State: Replica: 52d5abd2251e4b1dbdd3e60308ccd315, State: Running, Role: LEADER
I20260620 21:55:38.640584 22783 consensus_queue.cc:237] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } }
I20260620 21:55:38.643960 22308 catalog_manager.cc:5495] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52d5abd2251e4b1dbdd3e60308ccd315 (127.17.67.194). New cstate: current_term: 1 leader_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52d5abd2251e4b1dbdd3e60308ccd315" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 35209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:38.692734 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.691244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.17.67.194:35209)
0620 21:55:38.691376 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.691380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.691394 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:38.691554 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.691557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.691613 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.691682 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.692364 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.692367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.692369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.692378 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.692396 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.692403 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.692487 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.692535 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.692567 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:55:38.698694 22714 raft_consensus.cc:1261] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 1 FOLLOWER]: Refusing update from remote peer 52d5abd2251e4b1dbdd3e60308ccd315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:38.698696 22452 raft_consensus.cc:1261] T 177300352715495a94428caeb9fee55d P 96ed285608df4f97abd85f7554e4cf3c [term 1 FOLLOWER]: Refusing update from remote peer 52d5abd2251e4b1dbdd3e60308ccd315: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:38.699419 22783 consensus_queue.cc:1035] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bca8fadc0494f4ea25eb0bd1065547e" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 42589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:38.699638 22788 consensus_queue.cc:1035] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ed285608df4f97abd85f7554e4cf3c" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:38.708065 22792 mvcc.cc:205] Tried to move back new op lower bound from 7299041438502445056 to 7299041438274674688. Current Snapshot: MvccSnapshot[applied={T|T < 7299041438502445056}]
I20260620 21:55:38.716109 22308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:38.716285 22308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:38.719005 22308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a39ff2c75c466c9a84498ec267dad3: Sending DeleteTablet for 3 replicas of tablet 177300352715495a94428caeb9fee55d
I20260620 21:55:38.719779 22563 tablet_service.cc:1514] Processing DeleteTablet for tablet 177300352715495a94428caeb9fee55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:38 UTC) from {username='slave'} at 127.0.0.1:57704
I20260620 21:55:38.719952 22432 tablet_service.cc:1514] Processing DeleteTablet for tablet 177300352715495a94428caeb9fee55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:38 UTC) from {username='slave'} at 127.0.0.1:35422
I20260620 21:55:38.720309 22694 tablet_service.cc:1514] Processing DeleteTablet for tablet 177300352715495a94428caeb9fee55d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:38 UTC) from {username='slave'} at 127.0.0.1:40532
I20260620 21:55:38.720643 22801 tablet_replica.cc:330] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: stopping tablet replica
I20260620 21:55:38.720745 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22370
I20260620 21:55:38.721724 22803 tablet_replica.cc:330] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: stopping tablet replica
I20260620 21:55:38.722077 22803 raft_consensus.cc:2229] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:38.722076 22801 raft_consensus.cc:2229] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:38.722388 22803 raft_consensus.cc:2258] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:38.722499 22801 raft_consensus.cc:2258] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:38.724094 22801 ts_tablet_manager.cc:1905] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:38.727250 22801 ts_tablet_manager.cc:1918] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:38.727358 22801 log.cc:1196] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/177300352715495a94428caeb9fee55d
I20260620 21:55:38.727727 22801 ts_tablet_manager.cc:1939] T 177300352715495a94428caeb9fee55d P 52d5abd2251e4b1dbdd3e60308ccd315: Deleting consensus metadata
W20260620 21:55:38.728621 22295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:40689 (error 108)
I20260620 21:55:38.729249 22297 catalog_manager.cc:4841] TS 52d5abd2251e4b1dbdd3e60308ccd315 (127.17.67.194:35209): tablet 177300352715495a94428caeb9fee55d (table test-table [id=3cff61f8fad9437697d2c80f40e4ad22]) successfully deleted
I20260620 21:55:38.729300 22803 ts_tablet_manager.cc:1905] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:55:38.729924 22774 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:38.729615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40689 (local address 127.0.0.1:35436)
0620 21:55:38.729739 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:38.729821 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40689: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 21:55:38.730108 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22501
W20260620 21:55:38.731360 22295 catalog_manager.cc:4584] TS 96ed285608df4f97abd85f7554e4cf3c (127.17.67.193:40689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 177300352715495a94428caeb9fee55d: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40689: connect: Connection reset by peer (error 104)
I20260620 21:55:38.732614 22803 ts_tablet_manager.cc:1918] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:38.732695 22803 log.cc:1196] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/177300352715495a94428caeb9fee55d
I20260620 21:55:38.732977 22803 ts_tablet_manager.cc:1939] T 177300352715495a94428caeb9fee55d P 9bca8fadc0494f4ea25eb0bd1065547e: Deleting consensus metadata
I20260620 21:55:38.734205 22295 catalog_manager.cc:4841] TS 9bca8fadc0494f4ea25eb0bd1065547e (127.17.67.195:42589): tablet 177300352715495a94428caeb9fee55d (table test-table [id=3cff61f8fad9437697d2c80f40e4ad22]) successfully deleted
I20260620 21:55:38.737082 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22632
I20260620 21:55:38.742764 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22279
2026-06-20T21:55:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1250 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 21:55:38.766678 17679 mini_oidc.cc:152] Starting JWKS server
I20260620 21:55:38.768303 17679 webserver.cc:460] Webserver started at https://127.0.0.1:33195/ using document root <none> and password file <none>
I20260620 21:55:38.768447 17679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:55:38.768683 17679 webserver.cc:460] Webserver started at http://127.0.0.1:43263/ using document root <none> and password file <none>
2026-06-20T21:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:38Z Disabled control of system clock
I20260620 21:55:38.778169 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36593
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:36593
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33195/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/testchainca.cert with env {}
W20260620 21:55:38.891889 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:38.892138 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:38.892175 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:38.896165 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:38.896222 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:38.896237 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:38.896253 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:38.896268 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:55:38.896284 22810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:55:38.896303 22810 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33195/jwks.json
I20260620 21:55:38.901429 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:36593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33195/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:38.902436 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:38.903528 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:38.911672 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:38.911691 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:38.911716 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:38.911921 22810 server_base.cc:1034] running on GCE node
I20260620 21:55:38.912608 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:38.913802 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:38.914988 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781992538914968 us; error 31 us; skew 500 ppm
I20260620 21:55:38.916900 22810 webserver.cc:460] Webserver started at http://127.17.67.254:40641/ using document root <none> and password file <none>
I20260620 21:55:38.917496 22810 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:38.917603 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:38.917832 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:38.919909 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "ae60f013f6244f21847fca843ef8a5fb"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "e198039ce22eaf8f0b050975815fb9b2"
server_key_iv: "470c3d287531f6a4ace4695c54a1f8a8"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.920501 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "ae60f013f6244f21847fca843ef8a5fb"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "e198039ce22eaf8f0b050975815fb9b2"
server_key_iv: "470c3d287531f6a4ace4695c54a1f8a8"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.924238 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:55:38.926774 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:38.928133 22810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:38.928292 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "ae60f013f6244f21847fca843ef8a5fb"
format_stamp: "Formatted at 2026-06-20 21:55:38 on dist-test-slave-cx8d"
server_key: "e198039ce22eaf8f0b050975815fb9b2"
server_key_iv: "470c3d287531f6a4ace4695c54a1f8a8"
server_key_version: "encryptionkey@0"
I20260620 21:55:38.928418 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:38.956931 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:38.957715 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:38.957939 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:38.964797 22810 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:36593
I20260620 21:55:38.964823 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:36593 every 8 connection(s)
I20260620 21:55:38.965866 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:38.969292 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:38.974124 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22810
I20260620 21:55:38.974289 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:38.974598 17679 external_mini_cluster.cc:1428] Setting key cbb229b6c80485a5212f235fab759398
I20260620 21:55:38.975178 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb: Bootstrap starting.
I20260620 21:55:38.977797 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:38.978652 22877 log.cc:824] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:38.980697 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb: No bootstrap required, opened a new log
I20260620 21:55:38.985271 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:38.976010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.17.67.254:36593)
0620 21:55:38.976705 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:38.976723 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:38.976776 (+ 53us) server_negotiation.cc:407] Connection header received
0620 21:55:38.977601 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:38.977624 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:38.978152 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:38.978514 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:38.978968 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:38.980310 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:38.981262 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:38.981963 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:38.982121 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:38.983034 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:38.983042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:38.983058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:38.983559 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:38.983627 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:38.983652 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:38.983749 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:38.983977 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 21:55:38.984299 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260620 21:55:38.985276 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } }
I20260620 21:55:38.985615 22877 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:38.985669 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae60f013f6244f21847fca843ef8a5fb, State: Initialized, Role: FOLLOWER
I20260620 21:55:38.986017 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } }
I20260620 21:55:38.986115 22877 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:38.986155 22877 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:38.986244 22877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:38.987330 22877 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } }
I20260620 21:55:38.987654 22877 leader_election.cc:302] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae60f013f6244f21847fca843ef8a5fb; no voters:
I20260620 21:55:38.988037 22877 leader_election.cc:288] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:38.988175 22882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:38.988300 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:38.988469 22882 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [term 1 LEADER]: Becoming Leader. State: Replica: ae60f013f6244f21847fca843ef8a5fb, State: Running, Role: LEADER
I20260620 21:55:38.988749 22882 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } }
I20260620 21:55:38.989324 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:38.989940 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae60f013f6244f21847fca843ef8a5fb. Latest consensus state: current_term: 1 leader_uuid: "ae60f013f6244f21847fca843ef8a5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } } }
I20260620 21:55:38.990058 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:38.990461 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae60f013f6244f21847fca843ef8a5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae60f013f6244f21847fca843ef8a5fb" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36593 } } }
I20260620 21:55:38.990617 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:38.990617 22888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:38.993824 22888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:38.999125 22888 catalog_manager.cc:1347] Generated new cluster ID: 71292b12fecc41f3a9c2e305574596c8
I20260620 21:55:38.999199 22888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:39.009483 22888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:39.010617 22888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:39.023603 22888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae60f013f6244f21847fca843ef8a5fb: Generated new TSK 0
I20260620 21:55:39.024497 22888 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:55:39.059437 22907 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 21:55:39.059561 22907 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:55:39.060173 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.042226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.17.67.254:36593)
0620 21:55:39.042383 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.042388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.042588 (+ 200us) server_negotiation.cc:407] Connection header received
0620 21:55:39.042795 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.042798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.042858 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.042973 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:55:39.043225 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.044058 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.045081 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.045431 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.045533 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.045913 (+ 380us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:55:39.059731 (+ 13818us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:55:39.060014 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 21:55:39.060253 22880 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 21:55:39.061285 22906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:39.042110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36593 (local address 127.0.0.1:35626)
0620 21:55:39.042501 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:39.042516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:39.042632 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:39.042994 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:39.042997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:39.043003 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:55:39.043120 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.043124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.044178 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.044181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.044968 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.044977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.045545 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.045548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.045677 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.045723 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:55:39.059957 (+ 14234us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:55:39.060159 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.67.254:36593: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.67.254:36593: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33195/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 21:55:39.062150 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22810
2026-06-20T21:55:39Z chronyd exiting
I20260620 21:55:39.099485 17679 test_util.cc:175] -----------------------------------------------
I20260620 21:55:39.099607 17679 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992513246304-17679-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 21:55:39.100713 17679 mini_oidc.cc:152] Starting JWKS server
I20260620 21:55:39.101627 17679 webserver.cc:460] Webserver started at https://127.0.0.1:40783/ using document root <none> and password file <none>
I20260620 21:55:39.101749 17679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:55:39.102041 17679 webserver.cc:460] Webserver started at http://127.0.0.1:41653/ using document root <none> and password file <none>
2026-06-20T21:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:39Z Disabled control of system clock
I20260620 21:55:39.110819 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36741
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:36741
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/testchainca.cert with env {}
W20260620 21:55:39.217867 22913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:39.218118 22913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:39.218156 22913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:39.222116 22913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:39.222172 22913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:39.222187 22913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:39.222203 22913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:39.222218 22913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:55:39.222258 22913 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:55:39.222287 22913 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40783/jwks.json
I20260620 21:55:39.227502 22913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:36741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:39.228513 22913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:39.229516 22913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:39.237864 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:39.237944 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:39.237812 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:39.238003 22913 server_base.cc:1034] running on GCE node
I20260620 21:55:39.238751 22913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:39.239879 22913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:39.241070 22913 hybrid_clock.cc:648] HybridClock initialized: now 1781992539241061 us; error 28 us; skew 500 ppm
I20260620 21:55:39.242941 22913 webserver.cc:460] Webserver started at http://127.17.67.254:35511/ using document root <none> and password file <none>
I20260620 21:55:39.243530 22913 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:39.243623 22913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:39.243850 22913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:39.245988 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "057f62c5f7f54348bbe7c42220abbe36"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "8c39a8cc84f4ce5025949b974e35e631"
server_key_iv: "2850b2c6e47997e8c315a8dfa9d29c65"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.246541 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "057f62c5f7f54348bbe7c42220abbe36"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "8c39a8cc84f4ce5025949b974e35e631"
server_key_iv: "2850b2c6e47997e8c315a8dfa9d29c65"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.250044 22913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:55:39.252331 22927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:39.253505 22913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:39.253659 22913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "057f62c5f7f54348bbe7c42220abbe36"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "8c39a8cc84f4ce5025949b974e35e631"
server_key_iv: "2850b2c6e47997e8c315a8dfa9d29c65"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.253768 22913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:39.280682 22913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:39.281399 22913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:39.281678 22913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:39.288857 22913 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:36741
I20260620 21:55:39.288870 22979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:36741 every 8 connection(s)
I20260620 21:55:39.289819 22913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:39.292951 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:39.296573 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 22913
I20260620 21:55:39.296716 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:39.297057 17679 external_mini_cluster.cc:1428] Setting key a61382e6aedee47a0fbeb1bd641fcc1b
I20260620 21:55:39.298926 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36: Bootstrap starting.
I20260620 21:55:39.301059 22980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:39.301844 22980 log.cc:824] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:39.303682 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36: No bootstrap required, opened a new log
I20260620 21:55:39.306361 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.298443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.17.67.254:36741)
0620 21:55:39.299106 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.299121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.299157 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:55:39.299777 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.299793 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.300173 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.300511 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:39.300926 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.302097 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.302976 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.303448 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.303624 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.304486 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:39.304491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:39.304503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:39.304866 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:39.304918 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:39.304937 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:39.305045 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:39.305204 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:55:39.305443 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260620 21:55:39.307013 22980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } }
I20260620 21:55:39.307265 22980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:39.307322 22980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057f62c5f7f54348bbe7c42220abbe36, State: Initialized, Role: FOLLOWER
I20260620 21:55:39.307754 22980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } }
I20260620 21:55:39.307902 22980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:39.307973 22980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:39.308089 22980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:39.309149 22980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } }
I20260620 21:55:39.309194 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:39.309494 22980 leader_election.cc:302] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 057f62c5f7f54348bbe7c42220abbe36; no voters:
I20260620 21:55:39.309854 22980 leader_election.cc:288] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:39.309953 22985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:39.310190 22985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [term 1 LEADER]: Becoming Leader. State: Replica: 057f62c5f7f54348bbe7c42220abbe36, State: Running, Role: LEADER
I20260620 21:55:39.310519 22985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } }
I20260620 21:55:39.311095 22980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:39.311719 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057f62c5f7f54348bbe7c42220abbe36. Latest consensus state: current_term: 1 leader_uuid: "057f62c5f7f54348bbe7c42220abbe36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } } }
I20260620 21:55:39.311932 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.312139 22987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "057f62c5f7f54348bbe7c42220abbe36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057f62c5f7f54348bbe7c42220abbe36" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36741 } } }
I20260620 21:55:39.312240 22987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.312496 22994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:39.315563 22994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:39.320974 22994 catalog_manager.cc:1347] Generated new cluster ID: 0e47bb0bfb3f4211856650b59c2ab862
I20260620 21:55:39.321040 22994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:39.336403 22994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:39.337636 22994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:39.346163 22994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 057f62c5f7f54348bbe7c42220abbe36: Generated new TSK 0
I20260620 21:55:39.346863 22994 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:55:39.375283 23010 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 21:55:39.375388 23010 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:55:39.376057 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.359583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.17.67.254:36741)
0620 21:55:39.359720 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.359724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.359861 (+ 137us) server_negotiation.cc:407] Connection header received
0620 21:55:39.359998 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.360002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.360061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.360177 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:55:39.360362 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.361183 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.362118 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.362478 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.362575 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.362864 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:55:39.375661 (+ 12797us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:55:39.375921 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 21:55:39.376123 22983 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 21:55:39.376147 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:39.359479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36741 (local address 127.0.0.1:36980)
0620 21:55:39.359772 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:39.359785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:39.359879 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:39.360176 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:39.360178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:39.360185 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:55:39.360269 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.360273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.361289 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.361291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.362005 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.362013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.362597 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.362600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.362659 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.362679 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:55:39.375836 (+ 13157us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:55:39.375976 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.67.254:36741: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260620 21:55:39.376559 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 22913
2026-06-20T21:55:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 21:55:39.415761 17679 mini_oidc.cc:152] Starting JWKS server
I20260620 21:55:39.416693 17679 webserver.cc:460] Webserver started at https://127.0.0.1:37313/ using document root <none> and password file <none>
I20260620 21:55:39.416808 17679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:55:39.417094 17679 webserver.cc:460] Webserver started at http://127.0.0.1:37421/ using document root <none> and password file <none>
2026-06-20T21:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:39Z Disabled control of system clock
I20260620 21:55:39.425879 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:34939
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:34939
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37313/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:55:39.533984 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:39.534257 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:39.534296 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:39.538507 23016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:39.538576 23016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:39.538591 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:39.538609 23016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:39.538625 23016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:55:39.538640 23016 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:55:39.538654 23016 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37313/jwks.json
I20260620 21:55:39.543463 23016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:34939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37313/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:39.544528 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:39.546005 23016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:39.554572 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:39.554709 23016 server_base.cc:1034] running on GCE node
W20260620 21:55:39.554867 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:39.554891 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:39.555437 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:39.556595 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:39.557789 23016 hybrid_clock.cc:648] HybridClock initialized: now 1781992539557749 us; error 53 us; skew 500 ppm
I20260620 21:55:39.559713 23016 webserver.cc:460] Webserver started at http://127.17.67.254:36263/ using document root <none> and password file <none>
I20260620 21:55:39.560343 23016 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:39.560441 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:39.560717 23016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:39.562829 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "e94fbf7b25684963ae41f0e5995d9253"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "3b91a7ba78c17726f9c426c2e221af3a"
server_key_iv: "267427eaf5387ee965916cb4187231ee"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.563398 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "e94fbf7b25684963ae41f0e5995d9253"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "3b91a7ba78c17726f9c426c2e221af3a"
server_key_iv: "267427eaf5387ee965916cb4187231ee"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.567294 23016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:39.569923 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:39.571388 23016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:55:39.571544 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "e94fbf7b25684963ae41f0e5995d9253"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "3b91a7ba78c17726f9c426c2e221af3a"
server_key_iv: "267427eaf5387ee965916cb4187231ee"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.571672 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:39.588469 23016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:39.589246 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:39.589465 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:39.596959 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:34939 every 8 connection(s)
I20260620 21:55:39.596967 23016 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:34939
I20260620 21:55:39.598037 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:39.601440 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23016
I20260620 21:55:39.601569 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:39.601609 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:39.601931 17679 external_mini_cluster.cc:1428] Setting key 11bb8d9052eb5d0cd3ee0ce8c80b8510
I20260620 21:55:39.608603 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253: Bootstrap starting.
I20260620 21:55:39.611243 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:39.611959 23083 log.cc:824] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:39.612006 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.603441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.17.67.254:34939)
0620 21:55:39.604105 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.604121 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.604159 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:55:39.604874 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.604893 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.605339 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.605749 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:39.606130 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.607150 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.608375 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.608929 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.609052 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.610038 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:39.610043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:39.610056 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:39.610457 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:39.610513 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:39.610533 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:39.610614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:39.610796 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:55:39.611170 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260620 21:55:39.614061 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253: No bootstrap required, opened a new log
I20260620 21:55:39.614975 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:39.617887 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } }
I20260620 21:55:39.618139 23083 raft_consensus.cc:374] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:39.618254 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94fbf7b25684963ae41f0e5995d9253, State: Initialized, Role: FOLLOWER
I20260620 21:55:39.618692 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } }
I20260620 21:55:39.618857 23083 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:39.618944 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:39.619060 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:39.619980 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } }
I20260620 21:55:39.620325 23083 leader_election.cc:302] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94fbf7b25684963ae41f0e5995d9253; no voters:
I20260620 21:55:39.620630 23083 leader_election.cc:288] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:39.620764 23088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:39.621037 23088 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [term 1 LEADER]: Becoming Leader. State: Replica: e94fbf7b25684963ae41f0e5995d9253, State: Running, Role: LEADER
I20260620 21:55:39.621351 23088 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } }
I20260620 21:55:39.621940 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:39.623169 23089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94fbf7b25684963ae41f0e5995d9253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } } }
I20260620 21:55:39.623342 23089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.623735 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94fbf7b25684963ae41f0e5995d9253. Latest consensus state: current_term: 1 leader_uuid: "e94fbf7b25684963ae41f0e5995d9253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94fbf7b25684963ae41f0e5995d9253" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 34939 } } }
I20260620 21:55:39.623842 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.624429 23098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:39.627036 23098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:39.632150 23098 catalog_manager.cc:1347] Generated new cluster ID: d7a47fea55b940e5ba336dff0649a472
I20260620 21:55:39.632218 23098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:39.646867 23098 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:39.648494 23098 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:39.655189 23098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94fbf7b25684963ae41f0e5995d9253: Generated new TSK 0
I20260620 21:55:39.655869 23098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:39.677212 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.672031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.17.67.254:34939)
0620 21:55:39.672161 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.672165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.672330 (+ 165us) server_negotiation.cc:407] Connection header received
0620 21:55:39.672512 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.672515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.672561 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.672675 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:55:39.672910 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.673462 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.674422 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.674684 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.674802 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.675140 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:55:39.676824 (+ 1684us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37313/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:55:39.677065 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37313/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:55:39.677285 23086 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37313/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 21:55:39.679157 23112 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:39.671958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:34939 (local address 127.0.0.1:34810)
0620 21:55:39.672245 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:39.672257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:39.672387 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:39.672675 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:39.672678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:39.672686 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:55:39.672793 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.672799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.673604 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.673608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.674292 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.674300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.674811 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.674815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.674916 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.674940 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:55:39.677013 (+ 2073us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37313/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:55:39.678970 (+ 1957us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.67.254:34939: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37313/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1734,"thread_start_us":104,"threads_started":1}
I20260620 21:55:39.679711 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23016
E20260620 21:55:39.688396 23113 webserver.cc:533] Webserver: sslize failed:
E20260620 21:55:39.688476 23113 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T21:55:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T21:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:39Z Disabled control of system clock
I20260620 21:55:39.728039 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44931
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44931
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/testchainca.cert with env {}
W20260620 21:55:39.841076 23117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:39.841349 23117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:39.841391 23117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:39.845516 23117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:39.845593 23117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:39.845610 23117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:39.845628 23117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:39.845642 23117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:55:39.845659 23117 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:55:39.845671 23117 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 21:55:39.851049 23117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:39.852053 23117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:39.853010 23117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:39.861539 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:39.861539 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:39.861671 23117 server_base.cc:1034] running on GCE node
W20260620 21:55:39.861589 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:39.862403 23117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:39.863446 23117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:39.864642 23117 hybrid_clock.cc:648] HybridClock initialized: now 1781992539864614 us; error 35 us; skew 500 ppm
I20260620 21:55:39.866477 23117 webserver.cc:460] Webserver started at http://127.17.67.254:43043/ using document root <none> and password file <none>
I20260620 21:55:39.867053 23117 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:39.867134 23117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:39.867378 23117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:39.869390 23117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "140e25f2d0a145f39d4365074a2cfc8e"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "cb40dae5325f2050d2d7487754e5b98c"
server_key_iv: "fd02025746065dd7ba01057d72a60bed"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.870014 23117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "140e25f2d0a145f39d4365074a2cfc8e"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "cb40dae5325f2050d2d7487754e5b98c"
server_key_iv: "fd02025746065dd7ba01057d72a60bed"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.874647 23117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:39.878340 23131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:39.879570 23117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:39.879698 23117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "140e25f2d0a145f39d4365074a2cfc8e"
format_stamp: "Formatted at 2026-06-20 21:55:39 on dist-test-slave-cx8d"
server_key: "cb40dae5325f2050d2d7487754e5b98c"
server_key_iv: "fd02025746065dd7ba01057d72a60bed"
server_key_version: "encryptionkey@0"
I20260620 21:55:39.879818 23117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:39.898670 23117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:39.899392 23117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:39.899603 23117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:39.906597 23117 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44931
I20260620 21:55:39.906638 23183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44931 every 8 connection(s)
I20260620 21:55:39.907578 23117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:39.910426 23184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:39.914145 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23117
I20260620 21:55:39.914243 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:39.914517 17679 external_mini_cluster.cc:1428] Setting key e16af0cf18750a7af8fd625d7ecf93a6
I20260620 21:55:39.916285 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e: Bootstrap starting.
I20260620 21:55:39.919235 23184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:39.920228 23184 log.cc:824] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:39.922446 23184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e: No bootstrap required, opened a new log
I20260620 21:55:39.925688 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.915882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.17.67.254:44931)
0620 21:55:39.916596 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.916614 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.916657 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:55:39.917493 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.917516 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.918155 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.918580 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:39.919060 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.920784 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.921754 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.922489 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.922657 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.923465 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:39.923480 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:39.923496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:39.924015 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:39.924088 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:39.924116 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:39.924229 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:39.924366 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 21:55:39.924658 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260620 21:55:39.927067 23184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } }
I20260620 21:55:39.927330 23184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:39.927416 23184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140e25f2d0a145f39d4365074a2cfc8e, State: Initialized, Role: FOLLOWER
I20260620 21:55:39.927953 23184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } }
I20260620 21:55:39.928125 23184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:39.928224 23184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:39.928333 23184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:39.928462 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:39.929620 23184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } }
I20260620 21:55:39.930001 23184 leader_election.cc:302] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 140e25f2d0a145f39d4365074a2cfc8e; no voters:
I20260620 21:55:39.930377 23184 leader_election.cc:288] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:39.930469 23189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:39.930693 23189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [term 1 LEADER]: Becoming Leader. State: Replica: 140e25f2d0a145f39d4365074a2cfc8e, State: Running, Role: LEADER
I20260620 21:55:39.930998 23189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } }
I20260620 21:55:39.931694 23184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:39.932813 23190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "140e25f2d0a145f39d4365074a2cfc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } } }
I20260620 21:55:39.932821 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 140e25f2d0a145f39d4365074a2cfc8e. Latest consensus state: current_term: 1 leader_uuid: "140e25f2d0a145f39d4365074a2cfc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140e25f2d0a145f39d4365074a2cfc8e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44931 } } }
I20260620 21:55:39.932986 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.933216 23190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:39.933400 23198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:39.936424 23198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:39.942320 23198 catalog_manager.cc:1347] Generated new cluster ID: 6c68ab1ccbc44e3a924f821ea588952b
I20260620 21:55:39.942392 23198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:39.957790 23198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:39.959004 23198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:39.967334 23198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 140e25f2d0a145f39d4365074a2cfc8e: Generated new TSK 0
I20260620 21:55:39.968168 23198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:40.868288 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:39.981028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.17.67.254:44931)
0620 21:55:39.981167 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:39.981174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:39.981314 (+ 140us) server_negotiation.cc:407] Connection header received
0620 21:55:39.981449 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:39.981453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:39.981508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:39.981655 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:55:39.981874 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.982700 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.983671 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.984039 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.984138 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.984505 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:55:40.867873 (+883368us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:55:40.868144 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39328: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 21:55:40.868366 23187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39328: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 21:55:40.868548 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:39.980927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44931 (local address 127.0.0.1:39328)
0620 21:55:39.981229 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:39.981241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:39.981337 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:39.981669 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:39.981671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:39.981678 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:55:39.981770 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.981775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.982809 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.982812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.983566 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:39.983575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:39.984175 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:39.984178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:39.984283 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:39.984317 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:55:40.868113 (+883796us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:55:40.868316 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.67.254:44931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":127,"threads_started":1}
I20260620 21:55:40.869155 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23117
2026-06-20T21:55:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1169 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:41 dist-test-slave-cx8d krb5kdc[23225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:41 dist-test-slave-cx8d krb5kdc[23225](info): set up 2 sockets
Jun 20 21:55:41 dist-test-slave-cx8d krb5kdc[23225](info): commencing operation
krb5kdc: starting...
W20260620 21:55:42.238358 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:42.360777 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41707
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:42.471503 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:42.471810 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:42.471901 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:42.475996 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:42.476086 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:42.476126 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:42.476184 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:42.476224 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:42.481405 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:42.482471 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:42.483486 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:42.491818 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:42.491878 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:42.492018 23238 server_base.cc:1034] running on GCE node
W20260620 21:55:42.491851 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:42.492617 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:42.493732 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:42.494925 23238 hybrid_clock.cc:648] HybridClock initialized: now 1781992542494897 us; error 46 us; skew 500 ppm
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:42.501183 23238 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:42.502115 23238 webserver.cc:460] Webserver started at http://127.17.67.254:41249/ using document root <none> and password file <none>
I20260620 21:55:42.502678 23238 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:42.502769 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:42.502997 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:42.505014 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "082b67c6ace24e55a9061ab48132f252"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "57adff9b4bb5c9d2ffc66e67fb048dfa"
server_key_iv: "6f18baff5731c2d1217704b5f3fe34ed"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.505613 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "082b67c6ace24e55a9061ab48132f252"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "57adff9b4bb5c9d2ffc66e67fb048dfa"
server_key_iv: "6f18baff5731c2d1217704b5f3fe34ed"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.509197 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:42.511662 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:42.512821 23238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:42.512953 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "082b67c6ace24e55a9061ab48132f252"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "57adff9b4bb5c9d2ffc66e67fb048dfa"
server_key_iv: "6f18baff5731c2d1217704b5f3fe34ed"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.513059 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:42.532867 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:42.536126 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:42.536353 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:42.543803 23238 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41707
I20260620 21:55:42.543807 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41707 every 8 connection(s)
I20260620 21:55:42.544855 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:42.546931 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23238
I20260620 21:55:42.547088 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:42.547354 17679 external_mini_cluster.cc:1428] Setting key 7d87d5b1619fe3f8d5ec444dd12ea7d0
I20260620 21:55:42.547839 23306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:42.554895 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252: Bootstrap starting.
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:42.557809 23306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:42.558790 23306 log.cc:824] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:42.561034 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252: No bootstrap required, opened a new log
I20260620 21:55:42.562661 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:42.548709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.17.67.254:41707)
0620 21:55:42.549120 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:42.549147 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:42.549183 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:55:42.549892 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:42.549911 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:42.550241 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:42.550523 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:42.551541 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.552595 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.553853 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.554420 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.554579 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:42.557468 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:42.557473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:42.557490 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:42.557522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:42.559487 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:42.559971 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:42.559976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:42.559982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:42.560110 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:42.560378 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:42.560381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:42.560382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:42.560921 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:42.561108 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:42.561418 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 21:55:42.561738 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":148,"threads_started":1}
I20260620 21:55:42.565305 23306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } }
I20260620 21:55:42.565527 23306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:42.565608 23306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082b67c6ace24e55a9061ab48132f252, State: Initialized, Role: FOLLOWER
I20260620 21:55:42.566038 23306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } }
I20260620 21:55:42.566157 23306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:42.566241 23306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:42.566342 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:42.567356 23306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } }
I20260620 21:55:42.567646 23306 leader_election.cc:302] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082b67c6ace24e55a9061ab48132f252; no voters:
I20260620 21:55:42.567947 23306 leader_election.cc:288] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:42.568020 23311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:42.568194 23311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [term 1 LEADER]: Becoming Leader. State: Replica: 082b67c6ace24e55a9061ab48132f252, State: Running, Role: LEADER
I20260620 21:55:42.568470 23311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } }
I20260620 21:55:42.569195 23306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:42.570189 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082b67c6ace24e55a9061ab48132f252. Latest consensus state: current_term: 1 leader_uuid: "082b67c6ace24e55a9061ab48132f252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } } }
I20260620 21:55:42.570333 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:42.570640 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082b67c6ace24e55a9061ab48132f252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082b67c6ace24e55a9061ab48132f252" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41707 } } }
I20260620 21:55:42.570719 23320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:42.570807 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:42.574076 23320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:42.579599 23320 catalog_manager.cc:1347] Generated new cluster ID: fe2d68f1f48c403e8f3cd0065ca47f66
I20260620 21:55:42.579674 23320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:42.590040 23320 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:42.591022 23320 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:42.599822 23320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082b67c6ace24e55a9061ab48132f252: Generated new TSK 0
I20260620 21:55:42.600539 23320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:42.633450 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:42.744959 23332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:42.745250 23332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:42.745296 23332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:42.749962 23332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:42.750030 23332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:42.750092 23332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:42.756276 23332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:42.757664 23332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:42.758826 23332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:42.768366 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:42.768368 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:42.768399 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:42.768771 23332 server_base.cc:1034] running on GCE node
I20260620 21:55:42.769158 23332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:42.769850 23332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:42.771018 23332 hybrid_clock.cc:648] HybridClock initialized: now 1781992542771002 us; error 27 us; skew 500 ppm
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:42.773883 23332 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:42.774883 23332 webserver.cc:460] Webserver started at http://127.17.67.193:40041/ using document root <none> and password file <none>
I20260620 21:55:42.775439 23332 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:42.775499 23332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:42.775704 23332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:42.777729 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "16f5119fcf8941c981a73fe96dfc5aa4"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "0c36465baf6c2cc6f8e853e04ba4175b"
server_key_iv: "76af01a48bc486adfa9b738725ea1a7a"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.778292 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "16f5119fcf8941c981a73fe96dfc5aa4"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "0c36465baf6c2cc6f8e853e04ba4175b"
server_key_iv: "76af01a48bc486adfa9b738725ea1a7a"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.781984 23332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:42.784440 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:42.785823 23332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:42.785960 23332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "16f5119fcf8941c981a73fe96dfc5aa4"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "0c36465baf6c2cc6f8e853e04ba4175b"
server_key_iv: "76af01a48bc486adfa9b738725ea1a7a"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.786072 23332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:42.799345 23332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:42.802526 23332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:42.802740 23332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:42.803340 23332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:42.804153 23332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:42.804234 23332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:42.804325 23332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:42.804375 23332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:42.813715 23332 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:43119
I20260620 21:55:42.813742 23460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:43119 every 8 connection(s)
I20260620 21:55:42.814670 23332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:42.819664 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23332
I20260620 21:55:42.819872 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:42.820171 17679 external_mini_cluster.cc:1428] Setting key 261c6c71854606ecd2c279ca618e3d71
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:42.830138 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:42.816577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:50569 (local address 127.17.67.254:41707)
0620 21:55:42.816710 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:42.816714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:42.817597 (+ 883us) server_negotiation.cc:407] Connection header received
0620 21:55:42.818962 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:42.818966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:42.819020 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:42.819129 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:42.820499 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.821212 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.823080 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.823444 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.823563 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:42.826469 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:42.826472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:42.826474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:42.826495 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:42.827923 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:42.828554 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:42.828556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:42.828558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:42.828627 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:42.829004 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:42.829006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:42.829007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:42.829163 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:42.829233 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:42.829922 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 21:55:42.830019 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:42.831115 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:42.817022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41707 (local address 127.17.67.193:50569)
0620 21:55:42.817447 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:42.817489 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:42.818740 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:42.819309 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:42.819316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:42.819690 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:42.820352 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:42.820369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.821377 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.821384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:42.822934 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:42.822951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:42.823599 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:42.823602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:42.823757 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:42.824408 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:42.824423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:42.826297 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:42.828042 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:42.828049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:42.828068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:42.828441 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:42.828719 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:42.828722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:42.828724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:42.828906 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:42.829276 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:42.829285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:42.829651 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 21:55:42.829872 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 21:55:42.830207 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260620 21:55:42.832340 23461 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:55:42.832640 23461 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:42.833272 23461 heartbeater.cc:511] Master 127.17.67.254:41707 requested a full tablet report, sending...
I20260620 21:55:42.835127 23270 ts_manager.cc:194] Registered new tserver with Master: 16f5119fcf8941c981a73fe96dfc5aa4 (127.17.67.193:43119)
I20260620 21:55:42.836290 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:50569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:42.849124 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:42.961464 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:42.961809 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:42.961899 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:42.966650 23466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:42.966743 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:42.966856 23466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:42.972946 23466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:42.974190 23466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:42.975333 23466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:42.985381 23474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:42.985368 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:42.985374 23472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:42.985707 23466 server_base.cc:1034] running on GCE node
I20260620 21:55:42.986174 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:42.986837 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:42.988030 23466 hybrid_clock.cc:648] HybridClock initialized: now 1781992542988013 us; error 31 us; skew 500 ppm
Jun 20 21:55:42 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:42.991006 23466 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:42.991907 23466 webserver.cc:460] Webserver started at http://127.17.67.194:38511/ using document root <none> and password file <none>
I20260620 21:55:42.992458 23466 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:42.992516 23466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:42.992725 23466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:42.994870 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "879467da204d4b31af3183ad4d51f7ee"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "44f09a509d5ef8be9242cd6c5a48bdf9"
server_key_iv: "094adefe968b58967c8321e9962b6dde"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.995441 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "879467da204d4b31af3183ad4d51f7ee"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "44f09a509d5ef8be9242cd6c5a48bdf9"
server_key_iv: "094adefe968b58967c8321e9962b6dde"
server_key_version: "encryptionkey@0"
I20260620 21:55:42.999251 23466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:43.001863 23481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.003186 23466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:43.003291 23466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "879467da204d4b31af3183ad4d51f7ee"
format_stamp: "Formatted at 2026-06-20 21:55:42 on dist-test-slave-cx8d"
server_key: "44f09a509d5ef8be9242cd6c5a48bdf9"
server_key_iv: "094adefe968b58967c8321e9962b6dde"
server_key_version: "encryptionkey@0"
I20260620 21:55:43.003444 23466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:43.020339 23466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:43.023914 23466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:43.024127 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:43.024757 23466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:43.025703 23466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:43.025753 23466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.025815 23466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:43.025848 23466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.035317 23466 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:42677
I20260620 21:55:43.035353 23594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:42677 every 8 connection(s)
I20260620 21:55:43.036223 23466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:43.045861 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23466
I20260620 21:55:43.046000 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992542, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:43.046334 17679 external_mini_cluster.cc:1428] Setting key 6edab07ab774d294b868e746706297d3
I20260620 21:55:43.050633 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.038190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:42753 (local address 127.17.67.254:41707)
0620 21:55:43.038372 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.038377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.039253 (+ 876us) server_negotiation.cc:407] Connection header received
0620 21:55:43.040592 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.040596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.040660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.040745 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.042056 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.042621 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.043848 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.044074 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.044169 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.046896 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:43.046899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:43.046901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:43.046926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:43.048514 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.049039 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.049043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.049045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.049127 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.049476 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.049481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.049482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.049718 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:43.049833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:43.050342 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 21:55:43.050467 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 21:55:43.051558 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.038572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41707 (local address 127.17.67.194:42753)
0620 21:55:43.039063 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:43.039107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:43.040373 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:43.040909 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:43.040916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:43.041292 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:43.041905 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:43.041920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.042757 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.042760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:43.043655 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:43.043665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.044245 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.044269 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:43.044416 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.045066 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:43.045083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:43.046690 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:43.048646 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:43.048652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:43.048669 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:43.048935 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:43.049228 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:43.049231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:43.049233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:43.049383 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:43.049834 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:43.049842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:43.050092 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:55:43.050298 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 21:55:43.050633 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260620 21:55:43.052899 23595 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41707
I20260620 21:55:43.053145 23595 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:43.053702 23595 heartbeater.cc:511] Master 127.17.67.254:41707 requested a full tablet report, sending...
I20260620 21:55:43.054808 23270 ts_manager.cc:194] Registered new tserver with Master: 879467da204d4b31af3183ad4d51f7ee (127.17.67.194:42677)
I20260620 21:55:43.055397 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:42753
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:43.074926 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:43.189179 23600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:43.189435 23600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:43.189518 23600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:43.193902 23600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:43.193962 23600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:43.194049 23600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:43.199361 23600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:43.200541 23600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:43.201612 23600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:43.211133 23606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:43.211145 23605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:43.211309 23600 server_base.cc:1034] running on GCE node
W20260620 21:55:43.211155 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:43.211897 23600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:43.212530 23600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:43.213739 23600 hybrid_clock.cc:648] HybridClock initialized: now 1781992543213707 us; error 48 us; skew 500 ppm
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.216650 23600 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:43.217609 23600 webserver.cc:460] Webserver started at http://127.17.67.195:44907/ using document root <none> and password file <none>
I20260620 21:55:43.218168 23600 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:43.218230 23600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:43.218395 23600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:43.220388 23600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "251374069cfb48c6a41e193984dffa6c"
format_stamp: "Formatted at 2026-06-20 21:55:43 on dist-test-slave-cx8d"
server_key: "ba8fb00da82274e2987bbaba39ff5789"
server_key_iv: "8931bd45bc562903d3aba43a42a38f8e"
server_key_version: "encryptionkey@0"
I20260620 21:55:43.220935 23600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "251374069cfb48c6a41e193984dffa6c"
format_stamp: "Formatted at 2026-06-20 21:55:43 on dist-test-slave-cx8d"
server_key: "ba8fb00da82274e2987bbaba39ff5789"
server_key_iv: "8931bd45bc562903d3aba43a42a38f8e"
server_key_version: "encryptionkey@0"
I20260620 21:55:43.224736 23600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:43.227360 23615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.228641 23600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:43.228740 23600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "251374069cfb48c6a41e193984dffa6c"
format_stamp: "Formatted at 2026-06-20 21:55:43 on dist-test-slave-cx8d"
server_key: "ba8fb00da82274e2987bbaba39ff5789"
server_key_iv: "8931bd45bc562903d3aba43a42a38f8e"
server_key_version: "encryptionkey@0"
I20260620 21:55:43.228886 23600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:43.247314 23600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:43.250763 23600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:43.250988 23600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:43.251606 23600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:43.252432 23600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:43.252478 23600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.252542 23600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:43.252575 23600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:43.261211 23600 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:42637
I20260620 21:55:43.261286 23728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:42637 every 8 connection(s)
I20260620 21:55:43.262209 23600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:43.271667 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23600
I20260620 21:55:43.271814 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:43.272083 17679 external_mini_cluster.cc:1428] Setting key 90a59a2782085ec8b251909013d57da3
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:43.276502 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.264138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:57489 (local address 127.17.67.254:41707)
0620 21:55:43.264330 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.264335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.265163 (+ 828us) server_negotiation.cc:407] Connection header received
0620 21:55:43.266614 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.266617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.266674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.266788 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.267945 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.268478 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.269722 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.270007 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.270116 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.272851 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:43.272854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:43.272856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:43.272900 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:43.274541 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.275182 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.275186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.275188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.275279 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.275561 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.275563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.275564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.275724 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:43.275806 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:43.276299 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 21:55:43.276390 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 21:55:43.277429 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.264525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41707 (local address 127.17.67.195:57489)
0620 21:55:43.265001 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:43.265044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:43.266398 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:43.266958 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:43.266965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:43.267315 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:43.267796 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:43.267809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.268605 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.268609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:43.269541 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:43.269550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.270152 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.270156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:43.270333 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.270997 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:43.271013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:43.272671 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:43.274670 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:43.274714 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:43.274727 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:43.275046 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:43.275356 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:43.275358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:43.275359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:43.275486 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:43.275843 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:43.275848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:43.276080 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 21:55:43.276264 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 21:55:43.276519 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":103,"threads_started":1}
I20260620 21:55:43.278554 23729 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41707
I20260620 21:55:43.278798 23729 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:43.279280 23729 heartbeater.cc:511] Master 127.17.67.254:41707 requested a full tablet report, sending...
I20260620 21:55:43.280220 23270 ts_manager.cc:194] Registered new tserver with Master: 251374069cfb48c6a41e193984dffa6c (127.17.67.195:42637)
I20260620 21:55:43.280783 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:57489
I20260620 21:55:43.286474 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/test-admin.keytab.
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.302572 17679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 21:55:43.305528 23736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:43.304041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.17.67.193:43119)
0620 21:55:43.304340 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.304356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.304380 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:55:43.304434 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.304439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.304643 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.304778 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.305274 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46554: BlockingRecv error: recv got EOF from 127.0.0.1:46554 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
W20260620 21:55:43.305641 17679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.306923 17679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.308001 17679 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:43.318329 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.309075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.17.67.193:43119)
0620 21:55:43.309238 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.309241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.309314 (+ 73us) server_negotiation.cc:407] Connection header received
0620 21:55:43.309417 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.309420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.309460 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.309531 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.310096 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.310887 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.311786 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.312167 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.312260 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.314678 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:43.314687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:43.314701 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:43.314726 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:43.316611 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.317020 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.317025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.317027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.317102 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.317377 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.317379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.317381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.317674 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:43.317836 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:43.317998 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 21:55:43.318161 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 21:55:43.322110 23736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:43.321120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.17.67.193:43119)
0620 21:55:43.321289 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.321292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.321306 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:43.321435 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.321437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.321477 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.321543 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.322004 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46562: BlockingRecv error: recv got EOF from 127.0.0.1:46562 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 21:55:43.322152 17679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.323228 17679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.324167 17679 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:43.332996 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.324995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.17.67.193:43119)
0620 21:55:43.325121 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.325125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.325230 (+ 105us) server_negotiation.cc:407] Connection header received
0620 21:55:43.325369 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.325372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.325410 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.325500 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.326015 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.326546 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.327357 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.327602 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.327714 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.329963 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:43.329967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:43.329970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:43.329994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:43.331430 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.331869 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.331875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.331877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.331954 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.332222 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.332227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.332228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.332394 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:43.332552 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:43.332742 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:55:43.332843 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 21:55:43.335949 23736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:43.334795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.17.67.193:43119)
0620 21:55:43.334908 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.334913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.334966 (+ 53us) server_negotiation.cc:407] Connection header received
0620 21:55:43.335120 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.335124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.335164 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.335281 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.335850 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46566: BlockingRecv error: recv got EOF from 127.0.0.1:46566 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 21:55:43.336071 17679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992513246304-17679-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.337090 17679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:43.338083 17679 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:55:43.347051 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:43.338916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.17.67.193:43119)
0620 21:55:43.339027 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:43.339030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:43.339216 (+ 186us) server_negotiation.cc:407] Connection header received
0620 21:55:43.339364 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:43.339367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:43.339407 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:43.339497 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:43.339996 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.340499 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.341293 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:43.341525 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:43.341685 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:43.343928 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:43.343931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:43.343934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:43.343959 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:43.345404 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.345885 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.345890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.345891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.345971 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:43.346245 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:43.346250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:43.346251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:43.346416 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:43.346551 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:43.346792 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:55:43.346891 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:55:43.348062 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23332
I20260620 21:55:43.353808 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23466
I20260620 21:55:43.359129 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23600
I20260620 21:55:43.364331 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23238
2026-06-20T21:55:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2251 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23749](info): set up 2 sockets
Jun 20 21:55:43 dist-test-slave-cx8d krb5kdc[23749](info): commencing operation
krb5kdc: starting...
W20260620 21:55:44.421984 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:44 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab.
Entry for principal oryx/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab.
I20260620 21:55:44.544386 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45653
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab
--principal=oryx/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:45653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:44.654767 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:44.655053 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:44.655100 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:44.659463 23762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:44.659520 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:44.659540 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:44.659556 23762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:44.659572 23762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:44.664942 23762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:45653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab
--principal=oryx/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:44.666056 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:44.667192 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:44.675971 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:44.676002 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:44.675978 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:44.676271 23762 server_base.cc:1034] running on GCE node
I20260620 21:55:44.676920 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:44.678105 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:44.679330 23762 hybrid_clock.cc:648] HybridClock initialized: now 1781992544679298 us; error 40 us; skew 500 ppm
Jun 20 21:55:44 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:44.682175 23762 init.cc:377] Logged in from keytab as oryx/127.17.67.254@KRBTEST.COM (short username oryx)
I20260620 21:55:44.683046 23762 webserver.cc:460] Webserver started at http://127.17.67.254:43097/ using document root <none> and password file <none>
I20260620 21:55:44.683604 23762 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:44.683696 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:44.683928 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:44.686012 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "069736e4fde948ec8c8ae9c2cdb8b16a"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "4f5216df52d114afa93d05ea0a01ab9a"
server_key_iv: "023a4f953004ecee6570fa0bdd32fc8d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.686570 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "069736e4fde948ec8c8ae9c2cdb8b16a"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "4f5216df52d114afa93d05ea0a01ab9a"
server_key_iv: "023a4f953004ecee6570fa0bdd32fc8d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.690375 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:55:44.692838 23777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:44.694238 23762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:44.694386 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "069736e4fde948ec8c8ae9c2cdb8b16a"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "4f5216df52d114afa93d05ea0a01ab9a"
server_key_iv: "023a4f953004ecee6570fa0bdd32fc8d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.694496 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:44.718943 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:44.721946 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:44.722153 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:44.729380 23762 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:45653
I20260620 21:55:44.729681 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:45653 every 8 connection(s)
I20260620 21:55:44.730453 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:44.731149 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23762
I20260620 21:55:44.731283 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:44.731559 17679 external_mini_cluster.cc:1428] Setting key 65783cf578fb3e8583172fc0202b81b0
I20260620 21:55:44.733490 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:44.739010 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Bootstrap starting.
Jun 20 21:55:44 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:44.741569 23830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:44.742456 23830 log.cc:824] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:44.744706 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: No bootstrap required, opened a new log
I20260620 21:55:44.746207 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:44.732975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.17.67.254:45653)
0620 21:55:44.733402 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:44.733417 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:44.733451 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:55:44.734159 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:44.734176 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:44.734510 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:44.734852 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:44.735640 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:44.736629 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:44.737492 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:44.738028 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:44.738201 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:44.741029 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:44.741034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:44.741047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:44.741075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:44.743019 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:44.743559 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:44.743564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:44.743569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:44.743679 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:44.743977 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:44.743980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:44.743981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:44.744384 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:44.744516 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:44.744899 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 21:55:44.745218 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":149,"threads_started":1}
I20260620 21:55:44.748189 23830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } }
I20260620 21:55:44.748486 23830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:44.748574 23830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069736e4fde948ec8c8ae9c2cdb8b16a, State: Initialized, Role: FOLLOWER
I20260620 21:55:44.748947 23830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } }
I20260620 21:55:44.749058 23830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:44.749102 23830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:44.749169 23830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:44.750097 23830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } }
I20260620 21:55:44.750381 23830 leader_election.cc:302] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 069736e4fde948ec8c8ae9c2cdb8b16a; no voters:
I20260620 21:55:44.750628 23830 leader_election.cc:288] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:44.750799 23835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:44.751076 23835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 LEADER]: Becoming Leader. State: Replica: 069736e4fde948ec8c8ae9c2cdb8b16a, State: Running, Role: LEADER
I20260620 21:55:44.751384 23835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } }
I20260620 21:55:44.751852 23830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:44.753142 23837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069736e4fde948ec8c8ae9c2cdb8b16a. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } }
I20260620 21:55:44.753142 23836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } }
I20260620 21:55:44.753293 23837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:44.753293 23836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:44.753698 23845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:44.756553 23845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:44.762424 23845 catalog_manager.cc:1347] Generated new cluster ID: bd6ad43369024de2807a0ac9307a9ccf
I20260620 21:55:44.762499 23845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:44.770021 23845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:44.771270 23845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:44.786193 23845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Generated new TSK 0
I20260620 21:55:44.787017 23845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
I20260620 21:55:44.822724 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:44.940721 23856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:44.940980 23856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:44.941021 23856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:44.945041 23856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:44.945102 23856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:44.945158 23856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:44.950371 23856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:44.951407 23856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:44.952464 23856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:44.962080 23864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:44.962080 23861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:44.962332 23856 server_base.cc:1034] running on GCE node
W20260620 21:55:44.962080 23862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:44.962826 23856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:44.963429 23856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:44.964618 23856 hybrid_clock.cc:648] HybridClock initialized: now 1781992544964597 us; error 29 us; skew 500 ppm
Jun 20 21:55:44 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:44.967478 23856 init.cc:377] Logged in from keytab as oryx/127.17.67.193@KRBTEST.COM (short username oryx)
I20260620 21:55:44.968406 23856 webserver.cc:460] Webserver started at http://127.17.67.193:35993/ using document root <none> and password file <none>
I20260620 21:55:44.968943 23856 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:44.969000 23856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:44.969157 23856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:44.971185 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "cddfd4d29abf0660def0c0529ea95232"
server_key_iv: "6244379dbc2d9e24d98505852bce7c9d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.971678 23856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "cddfd4d29abf0660def0c0529ea95232"
server_key_iv: "6244379dbc2d9e24d98505852bce7c9d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.975487 23856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:44.977823 23871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:44.979156 23856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:44.979290 23856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "cddfd4d29abf0660def0c0529ea95232"
server_key_iv: "6244379dbc2d9e24d98505852bce7c9d"
server_key_version: "encryptionkey@0"
I20260620 21:55:44.979398 23856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:44.994800 23856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:44.998164 23856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:44.998366 23856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:44.999043 23856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:44.999853 23856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:44.999931 23856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:44.999995 23856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:45.000047 23856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.009210 23856 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:43027
I20260620 21:55:45.009290 23984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:43027 every 8 connection(s)
I20260620 21:55:45.010228 23856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:45.019261 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23856
I20260620 21:55:45.019372 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:45.019690 17679 external_mini_cluster.cc:1428] Setting key e7f5fef8b0952c4af4daea78b4837818
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:45.025713 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.012125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57671 (local address 127.17.67.254:45653)
0620 21:55:45.012265 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.012269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.013018 (+ 749us) server_negotiation.cc:407] Connection header received
0620 21:55:45.014548 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.014552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.014611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.014698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.015890 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.016479 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.017907 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.018147 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.018260 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.021059 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.021063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.021065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.021091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.023026 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.023820 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.023824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.023825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.023907 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.024366 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.024370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.024372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.024560 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.024650 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.025267 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.025434 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:55:45.026535 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.012424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:57671)
0620 21:55:45.012869 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.012912 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.014336 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.014885 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.014892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.015263 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.015733 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.015746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.016605 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.016615 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.017782 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.017792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.018302 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.018306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.018479 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.019004 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.019037 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.020887 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.023187 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.023198 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.023226 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.023658 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.024013 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.024017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.024022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.024229 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.024683 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.024688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.024981 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.025166 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.025445 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260620 21:55:45.027951 23985 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:45.028213 23985 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:45.028813 23985 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:45.030438 23794 ts_manager.cc:194] Registered new tserver with Master: 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:45.031555 23794 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:57671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
I20260620 21:55:45.048012 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:45.163615 23990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:45.163892 23990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:45.163935 23990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:45.168087 23990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:45.168161 23990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:45.168234 23990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:45.174584 23990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:45.175738 23990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:45.176819 23990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:45.187250 23998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.187294 23995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.187351 23990 server_base.cc:1034] running on GCE node
W20260620 21:55:45.187229 23996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.188122 23990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:45.188802 23990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:45.189973 23990 hybrid_clock.cc:648] HybridClock initialized: now 1781992545189954 us; error 31 us; skew 500 ppm
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:45.193043 23990 init.cc:377] Logged in from keytab as oryx/127.17.67.194@KRBTEST.COM (short username oryx)
I20260620 21:55:45.194029 23990 webserver.cc:460] Webserver started at http://127.17.67.194:46139/ using document root <none> and password file <none>
I20260620 21:55:45.194597 23990 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.194655 23990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.194871 23990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:45.196941 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "b2289f55d80d4afe89fdfcd0046df05e"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "883b9ccaa724ce82438c1d53b8d640b6"
server_key_iv: "2b67fcb2da7f80d29d7922024deadb8c"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.197613 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "b2289f55d80d4afe89fdfcd0046df05e"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "883b9ccaa724ce82438c1d53b8d640b6"
server_key_iv: "2b67fcb2da7f80d29d7922024deadb8c"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.201778 23990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:45.204324 24005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.205842 23990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:45.205991 23990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "b2289f55d80d4afe89fdfcd0046df05e"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "883b9ccaa724ce82438c1d53b8d640b6"
server_key_iv: "2b67fcb2da7f80d29d7922024deadb8c"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.206102 23990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:45.227437 23990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:45.230845 23990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:45.231002 23990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:45.231565 23990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:45.232372 23990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:45.232420 23990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.232460 23990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:45.232475 23990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.241518 23990 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40621
I20260620 21:55:45.241539 24118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40621 every 8 connection(s)
I20260620 21:55:45.242558 23990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:45.244535 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 23990
I20260620 21:55:45.244648 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:45.244987 17679 external_mini_cluster.cc:1428] Setting key a211b6e08d0ee4a869a6377992fc6a9c
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.195@KRBTEST.COM" created.
I20260620 21:55:45.259454 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.244538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:37967 (local address 127.17.67.254:45653)
0620 21:55:45.244692 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.244697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.245874 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 21:55:45.247786 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.247789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.247838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.247950 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.249532 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.250351 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.251714 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.251978 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.252100 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.255323 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.255328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.255335 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.255361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.256996 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.257645 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.257650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.257656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.257753 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.258150 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.258156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.258159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.258429 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.258541 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.259071 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.259227 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:55:45.260540 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.245132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:37967)
0620 21:55:45.245722 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.245776 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.247490 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.248178 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.248191 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.248816 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.249357 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.249372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.250517 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.250521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.251571 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.251582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.252187 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.252195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.252459 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.253189 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.253212 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.255131 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.257126 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.257133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.257151 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.257493 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.257867 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.257871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.257874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.258035 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.258536 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.258542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.258798 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.258967 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.259315 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":175,"threads_started":1}
I20260620 21:55:45.261981 24119 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:45.262279 24119 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:45.262825 24119 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
I20260620 21:55:45.264261 23794 ts_manager.cc:194] Registered new tserver with Master: b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621)
I20260620 21:55:45.264858 23794 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:37967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
I20260620 21:55:45.276381 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:45.398387 24124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:45.398689 24124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:45.398785 24124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:45.403430 24124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:45.403522 24124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:45.403621 24124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:45.409569 24124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:45.410830 24124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:45.411926 24124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:45.421751 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.421875 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.421759 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.422365 24124 server_base.cc:1034] running on GCE node
I20260620 21:55:45.422780 24124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:45.423439 24124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:45.424603 24124 hybrid_clock.cc:648] HybridClock initialized: now 1781992545424567 us; error 53 us; skew 500 ppm
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:45.427454 24124 init.cc:377] Logged in from keytab as oryx/127.17.67.195@KRBTEST.COM (short username oryx)
I20260620 21:55:45.428406 24124 webserver.cc:460] Webserver started at http://127.17.67.195:37847/ using document root <none> and password file <none>
I20260620 21:55:45.428929 24124 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.428987 24124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.429149 24124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:45.431198 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "c222d642bf5542489c06d694e543c0cb"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "fb5c8fa7f491108e6c94770019398795"
server_key_iv: "e0cc509c13789683f61825f2bd43188e"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.431727 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "c222d642bf5542489c06d694e543c0cb"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "fb5c8fa7f491108e6c94770019398795"
server_key_iv: "e0cc509c13789683f61825f2bd43188e"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.435516 24124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:55:45.438696 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.440064 24124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:45.440233 24124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "c222d642bf5542489c06d694e543c0cb"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "fb5c8fa7f491108e6c94770019398795"
server_key_iv: "e0cc509c13789683f61825f2bd43188e"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.440359 24124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:45.456027 24124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:45.459324 24124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:45.459479 24124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:45.460009 24124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:45.460784 24124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:45.460834 24124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.460907 24124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:45.460923 24124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.469297 24124 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33321
I20260620 21:55:45.469388 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33321 every 8 connection(s)
I20260620 21:55:45.470248 24124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:45.473294 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24124
I20260620 21:55:45.473426 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:45.473826 17679 external_mini_cluster.cc:1428] Setting key d176a58ddebb3aa446be5d2a3313adbf
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:45.485741 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.472109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:44161 (local address 127.17.67.254:45653)
0620 21:55:45.472286 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.472291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.473329 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 21:55:45.475700 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.475703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.475799 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.475904 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.477332 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.477910 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.479105 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.479341 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.479427 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.482116 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.482120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.482122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.482146 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.483685 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.484209 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.484211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.484212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.484290 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.484645 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.484648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.484649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.484836 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.484925 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.485426 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.485525 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:55:45.486588 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.472497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:44161)
0620 21:55:45.473128 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.473193 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.475417 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.476071 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.476084 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.476647 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.477173 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.477188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.478024 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.478027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.478991 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.479000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.479472 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.479475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.479625 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.480275 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.480297 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.481933 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.483793 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.483797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.483810 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.484129 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.484391 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.484394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.484395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.484528 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.484961 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.484966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.485214 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.485379 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.485707 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":125,"threads_started":1}
I20260620 21:55:45.488067 24253 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:45.488339 24253 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:45.488862 24253 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
I20260620 21:55:45.490057 23793 ts_manager.cc:194] Registered new tserver with Master: c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321)
I20260620 21:55:45.490689 23793 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:44161
I20260620 21:55:45.500519 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.67.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.253.keytab.
Entry for principal oryx/127.17.67.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.253.keytab.
I20260620 21:55:45.528503 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:37903
--webserver_interface=127.17.67.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.253.keytab
--principal=oryx/127.17.67.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:45653,127.17.67.253:37903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:45.642892 24258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:45.643162 24258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:45.643203 24258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:45.647428 24258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:45.647493 24258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:45.647513 24258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:45.647531 24258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:45.647545 24258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:45.653041 24258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:45653,127.17.67.253:37903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.253.keytab
--principal=oryx/127.17.67.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:37903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.67.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
W20260620 21:55:45.653303 24258 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.67.254:45653,127.17.67.253:37903'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 21:55:45.654311 24258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:45.655383 24258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:45.663882 24263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.663913 24264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.664084 24258 server_base.cc:1034] running on GCE node
W20260620 21:55:45.663882 24266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.664860 24258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:45.666023 24258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:45.667219 24258 hybrid_clock.cc:648] HybridClock initialized: now 1781992545667192 us; error 33 us; skew 500 ppm
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:45.669996 24258 init.cc:377] Logged in from keytab as oryx/127.17.67.253@KRBTEST.COM (short username oryx)
I20260620 21:55:45.670887 24258 webserver.cc:460] Webserver started at http://127.17.67.253:37107/ using document root <none> and password file <none>
I20260620 21:55:45.671473 24258 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.671579 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.671828 24258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:45.674676 24258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/instance:
uuid: "063c27a815f9403db7b66be7e7099f15"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0cd37d14deccbf44ad8e6cfba46b5bfd"
server_key_iv: "a0c49f09484b9e34a313a9f87b3971f6"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.675181 24258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal/instance:
uuid: "063c27a815f9403db7b66be7e7099f15"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0cd37d14deccbf44ad8e6cfba46b5bfd"
server_key_iv: "a0c49f09484b9e34a313a9f87b3971f6"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.678879 24258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:55:45.681546 24273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.682937 24258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:45.683064 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "063c27a815f9403db7b66be7e7099f15"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0cd37d14deccbf44ad8e6cfba46b5bfd"
server_key_iv: "a0c49f09484b9e34a313a9f87b3971f6"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.683167 24258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:45.699657 24258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:45.703114 24258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:45.703367 24258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:45.711191 24258 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.253:37903
I20260620 21:55:45.711545 24325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.253:37903 every 8 connection(s)
I20260620 21:55:45.712299 24258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
I20260620 21:55:45.714917 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24258
I20260620 21:55:45.715349 24326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:45.720106 24326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.253@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:45.731516 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.716444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.17.67.253:37903)
0620 21:55:45.716858 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.716875 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.716912 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:45.718036 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.718056 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.718407 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.718712 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.719597 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.720761 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.721903 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.722631 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.722783 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.725367 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.725373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.725386 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.725418 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.727730 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.728218 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.728224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.728230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.728365 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.728660 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.728663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.728664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.729067 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.729264 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.730014 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.730415 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260620 21:55:45.732357 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.722009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55670)
0620 21:55:45.722468 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.722515 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.722635 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.723318 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.723324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.723711 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.723841 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.723847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.724682 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.724685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.725636 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.725660 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.726261 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.726267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.726424 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.726833 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.726852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.728510 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.730548 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.730555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.730561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.730863 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.731177 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.731180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.731182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.731281 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.731766 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.731772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.731876 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.732009 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.732175 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":198,"threads_started":1}
I20260620 21:55:45.732378 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.722055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.17.67.254:45653)
0620 21:55:45.722270 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.722275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.722602 (+ 327us) server_negotiation.cc:407] Connection header received
0620 21:55:45.723101 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.723104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.723161 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.723268 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.723974 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.724528 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.725800 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.726140 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.726223 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.728647 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.728651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.728655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.728695 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.730435 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.730975 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.730979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.730980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.731068 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.731382 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.731385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.731387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.731617 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.731706 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.732104 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.732231 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:45.733585 24326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.253@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
I20260620 21:55:45.742539 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.734212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.17.67.253:37903)
0620 21:55:45.734328 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.734334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.734392 (+ 58us) server_negotiation.cc:407] Connection header received
0620 21:55:45.734519 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.734523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.734559 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.734658 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.735031 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.735575 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.736948 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.737201 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.737315 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.739457 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.739461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.739464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.739487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.740858 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.741282 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.741289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.741291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.741367 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.741676 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.741681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.741683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.741856 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.741966 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.742230 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.742400 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":42}
I20260620 21:55:45.742537 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.734103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55014)
0620 21:55:45.734328 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.734345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.734432 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.734659 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.734662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.734835 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.734918 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.734923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.735695 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.735702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.736813 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.736822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.737310 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.737313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.737404 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.737923 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.737945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.739320 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.740970 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.740976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.740980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.741182 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.741461 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.741466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.741469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.741552 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.741955 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.741958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.742041 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.742181 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.742340 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 21:55:45.745781 24326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Bootstrap starting.
I20260620 21:55:45.748035 24326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:45.748730 24326 log.cc:824] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:45.750703 24326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: No bootstrap required, opened a new log
I20260620 21:55:45.753859 24326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } }
I20260620 21:55:45.754129 24326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:45.754171 24326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063c27a815f9403db7b66be7e7099f15, State: Initialized, Role: FOLLOWER
I20260620 21:55:45.754622 24326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } }
I20260620 21:55:45.755168 24332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } }
I20260620 21:55:45.755322 24332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:45.755887 24326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:55:45.760444 24343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:45.760537 24343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:45.762864 24258 master_runner.cc:428] Detected that this master 063c27a815f9403db7b66be7e7099f15 is joining an existing cluster
I20260620 21:55:45.762934 24258 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.67.253:37903
I20260620 21:55:45.773010 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.764849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55672)
0620 21:55:45.765302 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.765315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.765398 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.765649 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.765651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.765865 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.765968 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.765973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.767035 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.767038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.767631 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.767639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.768253 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.768256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.768326 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.768859 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.768871 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.769233 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.771375 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.771378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.771380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.771617 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.771932 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.771935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.771937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.772023 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.772452 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.772455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.772543 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.772654 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.772811 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":107,"threads_started":1}
I20260620 21:55:45.773031 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.764904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.17.67.254:45653)
0620 21:55:45.765035 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.765041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.765396 (+ 355us) server_negotiation.cc:407] Connection header received
0620 21:55:45.765468 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.765471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.765524 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.765629 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.766084 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.766917 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.767766 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.768113 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.768217 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.769375 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.769379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.769383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.769410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.771270 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.771725 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.771729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.771731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.771820 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.772119 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.772123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.772125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.772323 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.772411 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.772730 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.772852 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:55:45.777971 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.764949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.17.67.253:37903)
0620 21:55:45.765046 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.765051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.769948 (+ 4897us) server_negotiation.cc:407] Connection header received
0620 21:55:45.770100 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.770105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.770186 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.770256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.771098 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.771771 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.772551 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.772860 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.772957 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.774392 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.774395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.774398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.774421 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.776041 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.776587 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.776593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.776596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.776676 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.777106 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.777109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.777111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.777275 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.777379 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.777658 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.777841 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":33}
I20260620 21:55:45.777983 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.765058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55030)
0620 21:55:45.769865 (+ 4807us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.769881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.769986 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.770412 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.770416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.770694 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.770980 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.770986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.771872 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.771876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.772441 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.772448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.773271 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.773277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.773365 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.773824 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.773837 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.774226 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.776297 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.776299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.776300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.776486 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.776908 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.776911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.776913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.777001 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.777377 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.777379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.777466 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.777617 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.777767 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4711,"thread_start_us":91,"threads_started":1}
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.254@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
I20260620 21:55:45.790259 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.779140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.17.67.253:37903)
0620 21:55:45.779238 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.779242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.779376 (+ 134us) server_negotiation.cc:407] Connection header received
0620 21:55:45.779541 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.779543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.779582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.779696 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.780436 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.781041 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.782303 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.782599 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.782747 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.785519 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.785524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.785532 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.785553 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.788208 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.788801 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.788804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.788806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.788917 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.789239 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.789242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.789243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.789467 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.789554 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.790038 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.790138 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:55:45.790282 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.778377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55046)
0620 21:55:45.779247 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.779301 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.779431 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.779738 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.779745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.780186 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.780313 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.780320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.781147 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.781150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.782182 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.782193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.782728 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.782732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.782876 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.783534 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.783549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.785383 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.788345 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.788355 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.788364 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.788672 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.789024 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.789028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.789030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.789125 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.789600 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.789607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.789754 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.789914 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.790114 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":94,"threads_started":1}
I20260620 21:55:45.791950 23793 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.67.253:37903: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "063c27a815f9403db7b66be7e7099f15"
member_type: NON_VOTER
last_known_addr {
host: "127.17.67.253"
port: 37903
}
attrs {
promote: true
}
}
dest_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a"
cas_config_opid_index: -1
I20260620 21:55:45.792654 23793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } }
I20260620 21:55:45.794597 24355 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 063c27a815f9403db7b66be7e7099f15 (127.17.67.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.17.67.252@KRBTEST.COM; defaulting to no policy
I20260620 21:55:45.795454 24300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:45.796317 24355 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.67.253:37903
I20260620 21:55:45.796571 24355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:45.796701 24355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:45.796811 24258 master.cc:537] Master@127.17.67.253:37903 shutting down...
I20260620 21:55:45.796875 24300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 21:55:45.797540 24354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.252@KRBTEST.COM" created.
W20260620 21:55:45.800781 23781 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 21:55:45.802099 23781 consensus_peers.cc:487] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a -> Peer 063c27a815f9403db7b66be7e7099f15 (127.17.67.253:37903): Couldn't send request to peer 063c27a815f9403db7b66be7e7099f15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:55:45.802700 24258 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:45.802989 24258 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:45.803090 24258 tablet_replica.cc:330] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: stopping tablet replica
I20260620 21:55:45.818327 24258 master.cc:552] Master@127.17.67.253:37903 shutdown complete.
I20260620 21:55:45.818392 24258 master_runner.cc:305] Clearing existing system tablet
I20260620 21:55:45.818638 24258 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.818701 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.820365 24258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:55:45.821264 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.821530 24258 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:45.821676 24258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:45.821719 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "063c27a815f9403db7b66be7e7099f15"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0cd37d14deccbf44ad8e6cfba46b5bfd"
server_key_iv: "a0c49f09484b9e34a313a9f87b3971f6"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.252.keytab.
Entry for principal oryx/127.17.67.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.252.keytab.
I20260620 21:55:45.823904 24258 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:45.824520 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:46315
--webserver_interface=127.17.67.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.252.keytab
--principal=oryx/127.17.67.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:45.826268 24258 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:55:45.826361 24258 log.cc:1196] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 21:55:45.826618 24258 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Deleting consensus metadata
I20260620 21:55:45.826797 24258 master_runner.cc:315] Copying system tablet from 127.17.67.254:45653
I20260620 21:55:45.827958 24258 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: tablet copy: Beginning tablet copy session from remote peer at address 127.17.67.254:45653
I20260620 21:55:45.837961 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.828551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55688)
0620 21:55:45.828884 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:45.828899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:45.829031 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:45.829327 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:45.829333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:45.829603 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:45.830010 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.830023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.831089 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.831096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.832197 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:45.832223 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.832845 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.832850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:45.832985 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.833594 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:45.833616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:45.834172 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:45.836275 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.836282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.836287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.836534 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.836879 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:45.836881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:45.836884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:45.836971 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:45.837371 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:45.837374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:45.837474 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 21:55:45.837608 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 21:55:45.837754 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260620 21:55:45.837991 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:45.828625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.17.67.254:45653)
0620 21:55:45.828757 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:45.828762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:45.828993 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:55:45.829136 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:45.829139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:45.829190 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:45.829282 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:45.830147 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.830963 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.832356 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:45.832716 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:45.832815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:45.834342 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:45.834346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:45.834349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:45.834378 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:45.836152 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.836633 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.836636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.836638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.836782 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:45.837063 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:45.837067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:45.837069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:45.837278 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:45.837372 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:45.837683 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 21:55:45.837834 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:55:45.838958 23814 tablet_copy_service.cc:140] P 069736e4fde948ec8c8ae9c2cdb8b16a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 063c27a815f9403db7b66be7e7099f15 ({username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:55688)
I20260620 21:55:45.839146 23814 tablet_copy_service.cc:161] P 069736e4fde948ec8c8ae9c2cdb8b16a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 063c27a815f9403db7b66be7e7099f15 at {username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:55688: session id = 063c27a815f9403db7b66be7e7099f15-00000000000000000000000000000000
I20260620 21:55:45.840909 23814 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:55:45.842684 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:55:45.845016 24258 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: tablet copy: Starting download of 0 data blocks...
I20260620 21:55:45.845204 24258 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: tablet copy: Starting download of 1 WAL segments...
I20260620 21:55:45.846969 24258 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:55:45.848878 23814 tablet_copy_service.cc:342] P 069736e4fde948ec8c8ae9c2cdb8b16a: Request end of tablet copy session 063c27a815f9403db7b66be7e7099f15-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:55688
I20260620 21:55:45.849011 23814 tablet_copy_service.cc:433] P 069736e4fde948ec8c8ae9c2cdb8b16a: ending tablet copy session 063c27a815f9403db7b66be7e7099f15-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 063c27a815f9403db7b66be7e7099f15
I20260620 21:55:45.856249 24258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:45.857999 24368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.858603 24369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.858753 24371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.859186 24258 server_base.cc:1034] running on GCE node
I20260620 21:55:45.859360 24258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:45.859666 24258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:45.860817 24258 hybrid_clock.cc:648] HybridClock initialized: now 1781992545860815 us; error 49 us; skew 500 ppm
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:45.862341 24258 init.cc:377] Logged in from keytab as oryx/127.17.67.253@KRBTEST.COM (short username oryx)
I20260620 21:55:45.862855 24258 webserver.cc:460] Webserver started at http://127.17.67.253:45533/ using document root <none> and password file <none>
I20260620 21:55:45.863044 24258 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.863085 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.864392 24258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:45.865437 24378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.865736 24258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:45.865804 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "063c27a815f9403db7b66be7e7099f15"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0cd37d14deccbf44ad8e6cfba46b5bfd"
server_key_iv: "a0c49f09484b9e34a313a9f87b3971f6"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.865875 24258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:45.885790 24258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:45.888057 24258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:45.892736 24258 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.253:37903
I20260620 21:55:45.892771 24430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.253:37903 every 8 connection(s)
I20260620 21:55:45.893311 24258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
I20260620 21:55:45.895226 24431 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:55:45.896327 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Bootstrap starting.
I20260620 21:55:45.909045 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:55:45.909628 24431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Bootstrap complete.
I20260620 21:55:45.910435 24431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } }
I20260620 21:55:45.910615 24431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 063c27a815f9403db7b66be7e7099f15, State: Initialized, Role: LEARNER
I20260620 21:55:45.910781 24431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } }
I20260620 21:55:45.911170 24434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:45.911305 24431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:45.911360 24434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:55:45.914481 24445 catalog_manager.cc:1259] Loaded cluster ID: bd6ad43369024de2807a0ac9307a9ccf
I20260620 21:55:45.914608 24445 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: loading cluster ID for follower catalog manager: success
I20260620 21:55:45.916818 24445 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: acquiring CA information for follower catalog manager: success
I20260620 21:55:45.918283 24445 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:55:45.957907 24361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:45.958190 24361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:45.958271 24361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:45.962353 24361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:45.962426 24361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:45.962462 24361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:45.962518 24361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:45.962572 24361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:45.968047 24361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.252.keytab
--principal=oryx/127.17.67.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.67.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:45.969095 24361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:45.970168 24361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:45.978579 24453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.978554 24450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:45.978571 24451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:45.979039 24361 server_base.cc:1034] running on GCE node
I20260620 21:55:45.979539 24361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:45.980649 24361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:45.981849 24361 hybrid_clock.cc:648] HybridClock initialized: now 1781992545981819 us; error 46 us; skew 500 ppm
Jun 20 21:55:45 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:45.984843 24361 init.cc:377] Logged in from keytab as oryx/127.17.67.252@KRBTEST.COM (short username oryx)
I20260620 21:55:45.985726 24361 webserver.cc:460] Webserver started at http://127.17.67.252:42057/ using document root <none> and password file <none>
I20260620 21:55:45.986243 24361 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:45.986302 24361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:45.986457 24361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:45.988387 24361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/instance:
uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0d15c64dd70be0724810135022872170"
server_key_iv: "b11a1b12579d42a532864d9ae5f3eb4d"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.988902 24361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal/instance:
uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0d15c64dd70be0724810135022872170"
server_key_iv: "b11a1b12579d42a532864d9ae5f3eb4d"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.992542 24361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 21:55:45.995203 24460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:45.996548 24361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:45.996647 24361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0d15c64dd70be0724810135022872170"
server_key_iv: "b11a1b12579d42a532864d9ae5f3eb4d"
server_key_version: "encryptionkey@0"
I20260620 21:55:45.996762 24361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.014983 24361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:46.017886 24361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:46.018033 24361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:46.025178 24361 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.252:46315
I20260620 21:55:46.025534 24512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.252:46315 every 8 connection(s)
I20260620 21:55:46.026252 24361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
I20260620 21:55:46.029143 24513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:46.032399 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24361
I20260620 21:55:46.033541 24513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:46.033941 23985 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.252@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
I20260620 21:55:46.047608 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.035777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.17.67.254:45653)
0620 21:55:46.035918 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.035930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.036360 (+ 430us) server_negotiation.cc:407] Connection header received
0620 21:55:46.036582 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.036585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.036625 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.036690 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.037608 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.038429 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.039902 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.040213 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.040320 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.043141 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.043148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.043153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.043188 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.045155 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.045918 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.045922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.045924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.046018 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.046411 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.046414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.046415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.046647 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.046751 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.047298 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.047440 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:55:46.048415 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.035640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55700)
0620 21:55:46.036207 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.036267 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.036460 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.036785 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.036791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.037272 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.037451 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.037461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.038570 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.038573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.039790 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.039799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.040388 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.040392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.040515 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.041050 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.041076 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.042974 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.045272 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.045279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.045286 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.045816 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.046112 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.046115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.046118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.046292 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.046776 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.046783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.047050 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.047232 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.047577 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":1,"spinlock_wait_cycles":5376,"thread_start_us":177,"threads_started":1}
I20260620 21:55:46.048417 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.034056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.17.67.252:46315)
0620 21:55:46.034542 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.034562 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.034609 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:55:46.035353 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.035375 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.035793 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.036125 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.037148 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.038561 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.039768 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.040300 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.040440 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.043882 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.043892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.043909 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.043940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.045926 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.046425 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.046431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.046432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.046504 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.046813 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.046815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.046816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.047067 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.047251 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.047475 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.047638 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":334,"thread_start_us":128,"threads_started":1}
I20260620 21:55:46.050066 24513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.252@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
I20260620 21:55:46.058612 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.050578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55056)
0620 21:55:46.050699 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.050713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.050784 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.051155 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.051157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.051292 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.051377 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.051381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.052355 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.052358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.052909 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.052917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.053459 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.053462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.053557 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.054046 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.054057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.055397 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.057211 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.057213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.057215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.057408 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.057713 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.057715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.057716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.057793 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.058231 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.058233 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.058310 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.058389 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.058490 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 21:55:46.058648 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.050692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.17.67.253:37903)
0620 21:55:46.050963 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.050966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.050978 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:55:46.051013 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.051015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.051050 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.051122 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.051468 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.052203 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.053026 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.053308 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.053424 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.055541 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.055543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.055545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.055568 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.057074 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.057499 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.057502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.057503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.057625 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.057935 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.057938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.057939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.058129 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.058215 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.058424 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.058519 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260620 21:55:46.059458 24513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992545, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.252@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
I20260620 21:55:46.066841 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.059913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.0.0.1:43210)
0620 21:55:46.060022 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.060037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.060111 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.060409 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.060411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.060557 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.060613 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.060617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.061330 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.061334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.061911 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.061919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.062363 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.062367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.062440 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.062826 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.062837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.063980 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.065385 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.065390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.065394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.065624 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.065888 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.065893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.065895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.065972 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.066358 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.066363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.066444 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.066542 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.066674 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 21:55:46.066857 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.059956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.17.67.252:46315)
0620 21:55:46.060071 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.060075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.060088 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:46.060258 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.060261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.060298 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.060389 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.060720 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.061223 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.062019 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.062261 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.062367 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.064109 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.064112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.064115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.064143 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.065276 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.065714 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.065717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.065718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.065786 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.066083 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.066088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.066089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.066246 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.066351 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.066597 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.066736 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":43}
I20260620 21:55:46.070152 24513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Bootstrap starting.
I20260620 21:55:46.072528 24513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:46.073272 24513 log.cc:824] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:46.075205 24513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: No bootstrap required, opened a new log
I20260620 21:55:46.078240 24513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } }
I20260620 21:55:46.078445 24513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:46.078480 24513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad32c7f8dfc4150a08bcf7ed845dca5, State: Initialized, Role: FOLLOWER
I20260620 21:55:46.078917 24513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } }
I20260620 21:55:46.079577 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } } }
I20260620 21:55:46.079715 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:46.080120 24513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:55:46.085260 24530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:46.085354 24530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:46.088923 24361 master_runner.cc:428] Detected that this master 0ad32c7f8dfc4150a08bcf7ed845dca5 is joining an existing cluster
I20260620 21:55:46.088981 24361 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.67.252:46315
I20260620 21:55:46.100870 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.092027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.17.67.253:37903)
0620 21:55:46.092150 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.092155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.092675 (+ 520us) server_negotiation.cc:407] Connection header received
0620 21:55:46.092829 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.092833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.092881 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.092930 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.093625 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.094219 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.095271 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.095626 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.095691 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.097005 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.097008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.097011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.097037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.098822 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.099409 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.099413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.099416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.099504 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.099935 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.099939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.099941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.100165 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.100250 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.100580 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.100692 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:46.100926 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.092262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55060)
0620 21:55:46.092602 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.092614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.092730 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.093057 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.093060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.093307 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.093473 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.093481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.094334 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.094336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.095139 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.095147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.095842 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.095845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.095929 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.096450 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.096470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.096857 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.098943 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.098947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.098949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.099278 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.099627 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.099630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.099632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.099793 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.100325 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.100328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.100415 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.100532 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.100666 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":23,"thread_start_us":103,"threads_started":1}
I20260620 21:55:46.101691 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23856
I20260620 21:55:46.102424 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.092430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.17.67.252:46315)
0620 21:55:46.092533 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.092537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.092583 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:55:46.092750 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.092756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.092803 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.092848 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.093610 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.094576 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.095601 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.096778 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.096875 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.098393 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.098397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.098400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.098424 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.100236 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.100840 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.100844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.100847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.101366 (+ 519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.101669 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.101673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.101674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.101891 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.101976 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.102238 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.102326 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":990,"server-negotiator.queue_time_us":43}
I20260620 21:55:46.102694 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.091602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.0.0.1:43224)
0620 21:55:46.092506 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.092520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.092641 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.092972 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.092974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.093245 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.093477 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.093482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.094686 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.094690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.095503 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.095510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.096908 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.096914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.097025 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.097760 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.097778 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.098247 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.100333 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.100337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.100339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.100574 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.101468 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.101471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.101473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.101554 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.102008 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.102013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.102092 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.102514 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.102596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":57,"thread_start_us":84,"threads_started":1}
I20260620 21:55:46.108858 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:43027
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:45653
--webserver_port=35993
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:46.134785 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.091112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55714)
0620 21:55:46.091941 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.091958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.092084 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.092885 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.092888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.093228 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.093905 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.093917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.095627 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.095630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.096546 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.096556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.097516 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.097519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.097639 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.130381 (+ 32742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.130399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.130818 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.132926 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.132930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.132933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.133177 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.133528 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.133531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.133533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.133668 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.134128 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.134131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.134258 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.134377 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.134559 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":97,"thread_start_us":115,"threads_started":1}
I20260620 21:55:46.134776 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.091224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.17.67.254:45653)
0620 21:55:46.091393 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.091398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.092565 (+ 1167us) server_negotiation.cc:407] Connection header received
0620 21:55:46.092614 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.092617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.092668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.092753 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.094720 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.095499 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.096979 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.097311 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.097405 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.131059 (+ 33654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.131063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.131065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.131097 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.132766 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.133313 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.133319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.133326 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.133411 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.133793 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.133799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.133802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.134019 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.134141 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.134457 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.134587 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992544, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.254@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
I20260620 21:55:46.148459 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.139058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.17.67.252:46315)
0620 21:55:46.139168 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.139173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.139187 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:46.139270 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.139273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.139330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.139415 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.139964 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.140513 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.141394 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.141725 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.141854 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.144494 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.144499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.144501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.144531 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.146540 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.147112 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.147117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.147119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.147212 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.147526 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.147530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.147532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.147746 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.147863 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.148147 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.148269 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:55:46.148954 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.138803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.0.0.1:43230)
0620 21:55:46.138952 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.138970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.139059 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.139556 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.139559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.139744 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.139837 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.139842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.140630 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.140636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.141261 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.141271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.141868 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.141872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.141974 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.142492 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.142509 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.144273 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.146681 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.146687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.146689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.146931 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.147316 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.147321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.147324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.147417 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.147852 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.147856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.147970 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.148091 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.148273 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 21:55:46.150168 23793 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.67.252:46315: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
member_type: NON_VOTER
last_known_addr {
host: "127.17.67.252"
port: 46315
}
attrs {
promote: true
}
}
dest_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a"
cas_config_opid_index: 5
I20260620 21:55:46.150790 23793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
I20260620 21:55:46.151975 24354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.153456 24354 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.67.252:46315
I20260620 21:55:46.154044 24487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:46.155182 24487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:55:46.155939 24540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:46.157021 24361 master.cc:537] Master@127.17.67.252:46315 shutting down...
I20260620 21:55:46.158063 24355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.158180 24355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.163905 24487 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
W20260620 21:55:46.164157 24487 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
I20260620 21:55:46.165992 24487 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 063c27a815f9403db7b66be7e7099f15 (127.17.67.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:46.168478 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.155885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:55064)
0620 21:55:46.156148 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.156164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.156255 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.163460 (+ 7205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.163464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.163493 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:46.163592 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.163598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.164945 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.164949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.166416 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.166432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.167864 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.167871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.167985 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.168057 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.168153 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.168271 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 21:55:46.168540 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069736e4fde948ec8c8ae9c2cdb8b16a. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:46.168650 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:55:46.168749 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:46.168834 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:55:46.168916 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } }
I20260620 21:55:46.168982 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 21:55:46.169178 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.156429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.17.67.253:37903)
0620 21:55:46.162983 (+ 6554us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.162989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.163012 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:55:46.163073 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.163077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.163151 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.163293 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:46.163717 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.164630 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.166670 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.167390 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.167511 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.167785 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.168927 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.169026 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6443}
I20260620 21:55:46.169687 24405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 LEARNER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:55:46.169683 24541 mvcc.cc:205] Tried to move back new op lower bound from 7299041463439892480 to 7299041463302529024. Current Snapshot: MvccSnapshot[applied={T|T < 7299041463336714240}]
I20260620 21:55:46.170208 24355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 21:55:46.172144 24405 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.174409 24434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069736e4fde948ec8c8ae9c2cdb8b16a. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.174543 24434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:55:46.174659 24434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: NON_VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.174757 24434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:55:46.178962 24355 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: attempting to promote NON_VOTER 063c27a815f9403db7b66be7e7099f15 to VOTER
I20260620 21:55:46.179441 24355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
I20260620 21:55:46.181064 24405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 LEARNER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 21:55:46.181735 24361 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 21:55:46.181717 24540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 21:55:46.182020 24361 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 21:55:46.182113 24361 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: stopping tablet replica
W20260620 21:55:46.182693 23779 consensus_peers.cc:487] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a -> Peer 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252:46315): Couldn't send request to peer 0ad32c7f8dfc4150a08bcf7ed845dca5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:55:46.183495 24540 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 063c27a815f9403db7b66be7e7099f15 (127.17.67.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.185180 24540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.185310 24540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.185453 24540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.186482 24405 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 063c27a815f9403db7b66be7e7099f15 (127.17.67.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.185539 24540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.188206 24549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.188350 24549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:46.198571 24361 master.cc:552] Master@127.17.67.252:46315 shutdown complete.
I20260620 21:55:46.198640 24361 master_runner.cc:305] Clearing existing system tablet
I20260620 21:55:46.198886 24361 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:46.198957 24361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:46.200445 24361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:46.201450 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.201793 24361 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.201923 24361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:46.201956 24361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0d15c64dd70be0724810135022872170"
server_key_iv: "b11a1b12579d42a532864d9ae5f3eb4d"
server_key_version: "encryptionkey@0"
I20260620 21:55:46.203771 24361 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:46.206291 24361 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:55:46.206383 24361 log.cc:1196] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 21:55:46.206667 24361 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Deleting consensus metadata
I20260620 21:55:46.206823 24361 master_runner.cc:315] Copying system tablet from 127.17.67.254:45653
I20260620 21:55:46.207953 24361 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: tablet copy: Beginning tablet copy session from remote peer at address 127.17.67.254:45653
I20260620 21:55:46.217883 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.208533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:55726)
0620 21:55:46.208817 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.208831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.208944 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.209243 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.209246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.209473 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.209604 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.209612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.210815 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.210819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.211764 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.211776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.212523 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.212528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.212627 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.213226 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.213241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.213743 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.216071 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.216074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.216077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.216346 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.216691 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.216694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.216697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.216808 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.217281 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.217290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.217395 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.217506 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.217691 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260620 21:55:46.217921 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.208662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.17.67.254:45653)
0620 21:55:46.208830 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.208836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.208921 (+ 85us) server_negotiation.cc:407] Connection header received
0620 21:55:46.209052 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.209057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.209119 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.209230 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.209746 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.210634 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.211987 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.212378 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.212499 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.213904 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.213908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.213911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.213941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.215944 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.216472 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.216479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.216483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.216585 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.216913 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.216917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.216918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.217171 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.217281 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.217619 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.217756 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:55:46.219048 23814 tablet_copy_service.cc:140] P 069736e4fde948ec8c8ae9c2cdb8b16a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ad32c7f8dfc4150a08bcf7ed845dca5 ({username='oryx', principal='oryx/127.17.67.252@KRBTEST.COM'} at 127.0.0.1:55726)
I20260620 21:55:46.219151 23814 tablet_copy_service.cc:161] P 069736e4fde948ec8c8ae9c2cdb8b16a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ad32c7f8dfc4150a08bcf7ed845dca5 at {username='oryx', principal='oryx/127.17.67.252@KRBTEST.COM'} at 127.0.0.1:55726: session id = 0ad32c7f8dfc4150a08bcf7ed845dca5-00000000000000000000000000000000
I20260620 21:55:46.220244 23814 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:55:46.221725 24361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:55:46.224848 24361 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: tablet copy: Starting download of 0 data blocks...
I20260620 21:55:46.225145 24361 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: tablet copy: Starting download of 1 WAL segments...
I20260620 21:55:46.226822 24361 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:55:46.228962 23814 tablet_copy_service.cc:342] P 069736e4fde948ec8c8ae9c2cdb8b16a: Request end of tablet copy session 0ad32c7f8dfc4150a08bcf7ed845dca5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.67.252@KRBTEST.COM'} at 127.0.0.1:55726
I20260620 21:55:46.229056 23814 tablet_copy_service.cc:433] P 069736e4fde948ec8c8ae9c2cdb8b16a: ending tablet copy session 0ad32c7f8dfc4150a08bcf7ed845dca5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ad32c7f8dfc4150a08bcf7ed845dca5
I20260620 21:55:46.236397 24361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:46.238185 24561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:46.238260 24558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.239110 24361 server_base.cc:1034] running on GCE node
W20260620 21:55:46.241783 24559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.244014 24361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:46.244364 24361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:46.245515 24361 hybrid_clock.cc:648] HybridClock initialized: now 1781992546245498 us; error 39 us; skew 500 ppm
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:46.247272 24361 init.cc:377] Logged in from keytab as oryx/127.17.67.252@KRBTEST.COM (short username oryx)
I20260620 21:55:46.247771 24361 webserver.cc:460] Webserver started at http://127.17.67.252:41217/ using document root <none> and password file <none>
I20260620 21:55:46.247969 24361 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:46.248018 24361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:46.249490 24361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:46.250645 24568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.250998 24361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:46.251082 24361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "0d15c64dd70be0724810135022872170"
server_key_iv: "b11a1b12579d42a532864d9ae5f3eb4d"
server_key_version: "encryptionkey@0"
I20260620 21:55:46.251166 24361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.264007 24361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:46.266324 24361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:46.267123 24119 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
W20260620 21:55:46.267968 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:46.268335 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:46.268414 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:55:46.271122 24361 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.252:46315
I20260620 21:55:46.271727 24361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
I20260620 21:55:46.272997 24623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.252:46315 every 8 connection(s)
I20260620 21:55:46.274647 24624 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:55:46.275650 24624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Bootstrap starting.
W20260620 21:55:46.278002 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:46.278069 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:46.278208 24539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:46.284437 24624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:55:46.284735 24624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: Bootstrap complete.
I20260620 21:55:46.285383 24624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
I20260620 21:55:46.285516 24624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ad32c7f8dfc4150a08bcf7ed845dca5, State: Initialized, Role: LEARNER
I20260620 21:55:46.285661 24624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } }
I20260620 21:55:46.286008 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.286170 24624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:46.286195 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: LEARNER
I20260620 21:55:46.288759 24638 catalog_manager.cc:1259] Loaded cluster ID: bd6ad43369024de2807a0ac9307a9ccf
I20260620 21:55:46.288879 24638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: loading cluster ID for follower catalog manager: success
I20260620 21:55:46.289700 24539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:43027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=35993
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:46.291079 24539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:46.291656 24638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: acquiring CA information for follower catalog manager: success
I20260620 21:55:46.292477 24539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:55:46.293116 24638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:55:46.304953 24641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.310508 24539 server_base.cc:1034] running on GCE node
W20260620 21:55:46.310806 24640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:46.311218 24643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.311796 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:46.312810 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:46.314019 24539 hybrid_clock.cc:648] HybridClock initialized: now 1781992546313987 us; error 46 us; skew 500 ppm
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:46.317811 24539 init.cc:377] Logged in from keytab as oryx/127.17.67.193@KRBTEST.COM (short username oryx)
I20260620 21:55:46.318760 24539 webserver.cc:460] Webserver started at http://127.17.67.193:35993/ using document root <none> and password file <none>
I20260620 21:55:46.319327 24539 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:46.319401 24539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:46.323483 24539 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:55:46.325659 24650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.327026 24539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:46.327169 24539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
format_stamp: "Formatted at 2026-06-20 21:55:44 on dist-test-slave-cx8d"
server_key: "cddfd4d29abf0660def0c0529ea95232"
server_key_iv: "6244379dbc2d9e24d98505852bce7c9d"
server_key_version: "encryptionkey@0"
I20260620 21:55:46.327721 24539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.374010 24539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:46.377449 24539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:46.377676 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:46.378270 24539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:46.379141 24539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:46.379211 24539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.379289 24539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:46.379339 24539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.389803 24539 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:43027
I20260620 21:55:46.389874 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:43027 every 8 connection(s)
I20260620 21:55:46.390854 24539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:46.391197 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24539
I20260620 21:55:46.391429 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23990
I20260620 21:55:46.398176 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:40621
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:45653
--webserver_port=46139
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
I20260620 21:55:46.411324 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.394889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:45665 (local address 127.17.67.253:37903)
0620 21:55:46.395062 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.395067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.395441 (+ 374us) server_negotiation.cc:407] Connection header received
0620 21:55:46.397064 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.397072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.397159 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.397226 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.399439 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400364 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402233 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.402622 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.406683 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.406690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.406693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.406727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.408462 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.409161 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.409167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.409169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.409260 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.409739 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.409742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.409744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.410008 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.410110 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.410936 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.411078 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 21:55:46.412638 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.394923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:37935 (local address 127.17.67.254:45653)
0620 21:55:46.395187 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.395193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.397557 (+ 2364us) server_negotiation.cc:407] Connection header received
0620 21:55:46.397802 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.397805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.397867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.397932 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.399553 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400400 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402209 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.402597 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402734 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.407880 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.407887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.407893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.407929 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.410415 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411022 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.411028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.411032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.411124 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411532 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.411537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.411541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.411803 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.411900 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.412373 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.412488 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260620 21:55:46.413111 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.392715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:54483 (local address 127.17.67.252:46315)
0620 21:55:46.393107 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.393112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.395367 (+ 2255us) server_negotiation.cc:407] Connection header received
0620 21:55:46.397108 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.397114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.397185 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.397259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.399601 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400281 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402198 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.402628 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402725 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.408973 (+ 6248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.408980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.408985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.409018 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.411080 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411633 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.411638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.411640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.411730 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.412039 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.412043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.412044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.412304 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.412405 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.412805 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.412934 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":93,"threads_started":1}
I20260620 21:55:46.413834 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.393517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.17.67.193:45665)
0620 21:55:46.395352 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.395365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.396798 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.397685 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.397696 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.398413 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.399256 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.399279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400527 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.400533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.402072 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.402083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.402943 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.403374 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.404280 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.404311 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.406428 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.408598 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.408605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.408623 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.409030 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.409379 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.409382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.409384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.409625 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.410198 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.410204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.410589 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.411841 (+ 1252us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.412293 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"mutex_wait_us":284,"spinlock_wait_cycles":30720,"thread_start_us":157,"threads_started":1}
I20260620 21:55:46.414063 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.394786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:37935)
0620 21:55:46.397464 (+ 2678us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.397482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.397679 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.398081 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.398084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.398524 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.399451 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.399462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400515 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.400518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.402048 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.402059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.402943 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.402947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.403366 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.404338 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.404350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.407676 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.410554 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.410560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.410563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.410892 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.411310 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.411315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.411411 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.411966 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.411971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.412081 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.412871 (+ 790us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.413011 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2613,"mutex_wait_us":432,"spinlock_wait_cycles":2688,"thread_start_us":76,"threads_started":1}
I20260620 21:55:46.414081 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.393253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.17.67.193:54483)
0620 21:55:46.395184 (+ 1931us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.395247 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.396818 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.397689 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.397697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.398411 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.399256 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.399279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.400426 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.400428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.402076 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.402083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.403031 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.403038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.403355 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.404490 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.404511 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.408794 (+ 4283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.411207 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.411212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.411480 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.411844 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.411847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.411849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.411936 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.412476 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.412479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.412578 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.413325 (+ 747us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.413759 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1710,"mutex_wait_us":620,"spinlock_wait_cycles":16896,"thread_start_us":145,"threads_started":1}
I20260620 21:55:46.415521 24766 heartbeater.cc:347] Connected to a master server at 127.17.67.253:37903
I20260620 21:55:46.415781 24766 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.416126 24764 heartbeater.cc:347] Connected to a master server at 127.17.67.252:46315
I20260620 21:55:46.416205 24764 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.416369 24766 heartbeater.cc:511] Master 127.17.67.253:37903 requested a full tablet report, sending...
I20260620 21:55:46.416368 24764 heartbeater.cc:511] Master 127.17.67.252:46315 requested a full tablet report, sending...
I20260620 21:55:46.416895 24765 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:46.416965 24765 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.417135 24765 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
I20260620 21:55:46.418026 24395 ts_manager.cc:194] Registered new tserver with Master: 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:46.418311 23793 ts_manager.cc:194] Re-registered known tserver with Master: 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:46.418952 24579 ts_manager.cc:194] Registered new tserver with Master: 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:46.419085 23793 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:37935
I20260620 21:55:46.492738 24253 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
W20260620 21:55:46.518397 24772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:46.518635 24772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:46.518676 24772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:46.522471 24772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:46.522516 24772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:46.522570 24772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:46.528002 24772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:40621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=46139
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:46.528925 24772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:46.529896 24772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:46.539352 24778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:46.539417 24777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.539588 24772 server_base.cc:1034] running on GCE node
W20260620 21:55:46.539354 24780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.540200 24772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:46.540860 24772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:46.542060 24772 hybrid_clock.cc:648] HybridClock initialized: now 1781992546542040 us; error 29 us; skew 500 ppm
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:46.545115 24772 init.cc:377] Logged in from keytab as oryx/127.17.67.194@KRBTEST.COM (short username oryx)
I20260620 21:55:46.546154 24772 webserver.cc:460] Webserver started at http://127.17.67.194:46139/ using document root <none> and password file <none>
I20260620 21:55:46.546700 24772 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:46.546828 24772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:46.550770 24772 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:55:46.553026 24787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.554315 24772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:55:46.554436 24772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "b2289f55d80d4afe89fdfcd0046df05e"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "883b9ccaa724ce82438c1d53b8d640b6"
server_key_iv: "2b67fcb2da7f80d29d7922024deadb8c"
server_key_version: "encryptionkey@0"
I20260620 21:55:46.554982 24772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.567288 24772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:46.570393 24772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:46.570595 24772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:46.571172 24772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:46.571986 24772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:46.572033 24772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.572098 24772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:46.572129 24772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.581094 24772 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40621
I20260620 21:55:46.581115 24900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40621 every 8 connection(s)
I20260620 21:55:46.582121 24772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:46.586666 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24772
I20260620 21:55:46.586879 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24124
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:46.593737 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:33321
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:45653
--webserver_port=37847
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
I20260620 21:55:46.599007 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.583838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:42173 (local address 127.17.67.254:45653)
0620 21:55:46.583967 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.583972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.584885 (+ 913us) server_negotiation.cc:407] Connection header received
0620 21:55:46.586159 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.586162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.586221 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.586300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.588084 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.588647 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.590143 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.590482 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.590573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.593828 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.593833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.593836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.593863 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.596402 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597076 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.597079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.597082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.597165 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597598 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.597602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.597603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.597841 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.597930 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.598711 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.598837 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:46.599304 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.585589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:49925 (local address 127.17.67.252:46315)
0620 21:55:46.585716 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.585719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.585731 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:55:46.586425 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.586428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.586508 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.586585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.590099 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.590952 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.592101 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.592428 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.592505 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.595381 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.595384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.595387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.595428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.597174 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597740 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.597747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.597751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.597843 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.598170 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.598176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.598180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.598416 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.598514 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.598936 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.599086 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:55:46.600018 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.584279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.17.67.194:44483)
0620 21:55:46.584735 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.584785 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.585820 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.586337 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.586346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.589454 (+ 3108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.589604 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.589611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.590631 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.590633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.591488 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.591501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.592214 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.592220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.592332 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.592866 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.592881 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.594288 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.596598 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.596600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.596603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.596958 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.597267 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.597272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.597465 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.598009 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.598011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.598417 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.598633 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.598952 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":30,"spinlock_wait_cycles":31232,"thread_start_us":138,"threads_started":1}
I20260620 21:55:46.600026 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.584279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:42173)
0620 21:55:46.584736 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.584786 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.585828 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.586338 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.586347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.586907 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.587940 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.587953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.588796 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.588799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.590012 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.590021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.590782 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.590786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.590973 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.591608 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.591629 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.593627 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.596503 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.596509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.596525 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.596958 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.597267 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.597272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.597463 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.598001 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.598007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.598417 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.599135 (+ 718us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.599256 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":96,"spinlock_wait_cycles":29184,"thread_start_us":131,"threads_started":1}
I20260620 21:55:46.600780 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.583854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:44483 (local address 127.17.67.253:37903)
0620 21:55:46.584008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.584013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.584885 (+ 872us) server_negotiation.cc:407] Connection header received
0620 21:55:46.586018 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.586022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.586077 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.586156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.589740 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.590487 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.591629 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.591980 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.592057 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.594448 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.594454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.594458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.594490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.596493 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597076 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.597080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.597082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.597165 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597601 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.597604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.597606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.597848 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.597930 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.600532 (+ 2602us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.600659 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:55:46.600847 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.585252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.17.67.194:49925)
0620 21:55:46.585507 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.585521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.585820 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.586737 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.586739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.589823 (+ 3084us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.589951 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.589958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.591115 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.591118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.591972 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.591981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.592690 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.592693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.592787 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.593383 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.593397 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.595247 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.597298 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.597309 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.597615 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.597962 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.597965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.597968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.598063 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.598579 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.598582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.598718 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.599931 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.600105 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
I20260620 21:55:46.601888 24903 heartbeater.cc:347] Connected to a master server at 127.17.67.253:37903
I20260620 21:55:46.601944 24902 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:46.602180 24901 heartbeater.cc:347] Connected to a master server at 127.17.67.252:46315
I20260620 21:55:46.602146 24903 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.602144 24902 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.602283 24901 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.602830 24901 heartbeater.cc:511] Master 127.17.67.252:46315 requested a full tablet report, sending...
I20260620 21:55:46.602828 24902 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
I20260620 21:55:46.602895 24903 heartbeater.cc:511] Master 127.17.67.253:37903 requested a full tablet report, sending...
I20260620 21:55:46.603951 24395 ts_manager.cc:194] Registered new tserver with Master: b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621)
I20260620 21:55:46.604029 24579 ts_manager.cc:194] Registered new tserver with Master: b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621)
I20260620 21:55:46.605432 23793 ts_manager.cc:194] Re-registered known tserver with Master: b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621)
I20260620 21:55:46.606040 23793 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:42173
I20260620 21:55:46.696060 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.692484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.0.0.1:42174)
0620 21:55:46.692764 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.692779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.692932 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.693214 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.693217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.693234 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:46.693367 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.693372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.694181 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.694184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.694976 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.694985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.695668 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.695671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.695758 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.695782 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.695847 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.695923 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260620 21:55:46.696058 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.692550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.17.67.252:46315)
0620 21:55:46.692687 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.692691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.692865 (+ 174us) server_negotiation.cc:407] Connection header received
0620 21:55:46.693044 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.693047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.693097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.693193 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:46.693476 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.694064 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.695099 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.695534 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.695650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.695818 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.695857 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.695923 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:46.696666 24598 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 21:55:46.696925 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069736e4fde948ec8c8ae9c2cdb8b16a. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.697065 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: LEARNER
W20260620 21:55:46.726692 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:46.727025 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:46.727101 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:46.731464 24908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:46.731544 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:46.731635 24908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:46.737345 24908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:33321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=37847
--tserver_master_addrs=127.17.67.254:45653,127.17.67.253:37903,127.17.67.252:46315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:46.738425 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:46.739441 24908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:46.749480 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:46.749611 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:46.749477 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:46.749984 24908 server_base.cc:1034] running on GCE node
I20260620 21:55:46.750451 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:46.751255 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:46.752456 24908 hybrid_clock.cc:648] HybridClock initialized: now 1781992546752416 us; error 55 us; skew 500 ppm
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:46.755527 24908 init.cc:377] Logged in from keytab as oryx/127.17.67.195@KRBTEST.COM (short username oryx)
I20260620 21:55:46.756397 24908 webserver.cc:460] Webserver started at http://127.17.67.195:37847/ using document root <none> and password file <none>
I20260620 21:55:46.756904 24908 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:46.756971 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:46.761104 24908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:46.763208 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.764430 24908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:46.764564 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "c222d642bf5542489c06d694e543c0cb"
format_stamp: "Formatted at 2026-06-20 21:55:45 on dist-test-slave-cx8d"
server_key: "fb5c8fa7f491108e6c94770019398795"
server_key_iv: "e0cc509c13789683f61825f2bd43188e"
server_key_version: "encryptionkey@0"
I20260620 21:55:46.765133 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:46.784353 24908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:46.787928 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:46.788127 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:46.788823 24908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:46.789738 24908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:46.789804 24908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.789866 24908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:46.789896 24908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:46.799837 24908 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33321
I20260620 21:55:46.799855 25037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33321 every 8 connection(s)
I20260620 21:55:46.800892 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:46.802017 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 24908
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.253@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.252@KRBTEST.COM
Jun 20 21:55:46 dist-test-slave-cx8d krb5kdc[23749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992546, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:46.820044 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.803054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:58701 (local address 127.17.67.252:46315)
0620 21:55:46.803290 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.803294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.804531 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 21:55:46.805989 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.805995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.806055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.806139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.808183 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809039 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.811985 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.812412 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.812504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.815702 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.815708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.815711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.815744 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.817528 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.818205 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.818210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.818215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.818298 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.818721 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.818724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.818726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.818961 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.819047 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.819749 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.819873 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260620 21:55:46.820996 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.802922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:56529 (local address 127.17.67.253:37903)
0620 21:55:46.803193 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.803199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.805831 (+ 2632us) server_negotiation.cc:407] Connection header received
0620 21:55:46.806353 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.806358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.806419 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.806477 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.808655 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809461 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.810977 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.811324 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.811396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.814909 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.814912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.814916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.814946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.817649 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.819209 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.819212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.819213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.819289 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.819564 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.819568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.819569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.819799 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.819874 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.820673 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.820816 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260620 21:55:46.821884 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.803429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.17.67.195:58701)
0620 21:55:46.804363 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.804426 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.805721 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.806574 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.806586 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.807192 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.808014 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.808034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809211 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.809215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.811832 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.811856 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.812689 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.812693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.812826 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.813418 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.813434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.815534 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.817685 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.817693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.817716 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.818083 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.818415 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.818420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.818422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.818607 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.819112 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.819118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.819471 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.820294 (+ 823us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.820665 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":1019,"thread_start_us":143,"threads_started":1}
I20260620 21:55:46.821993 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.804101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.17.67.254:45653)
0620 21:55:46.804223 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.804227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.805361 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 21:55:46.805546 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.805548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.805652 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.805755 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.806909 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.807806 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.809000 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809327 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.809399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.815489 (+ 6090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.815494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.815497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.815526 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.818741 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.820192 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.820199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.820205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.820309 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.820661 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.820666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.820668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.820887 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.820998 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.821669 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.821818 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:46.822222 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.804678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.17.67.253:37903)
0620 21:55:46.804913 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.804917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.806068 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 21:55:46.806846 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.806849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.806901 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.806961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.808976 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809741 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.813433 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.813880 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.813988 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.816431 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.816435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.816439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.816471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.818968 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.820653 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.820656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.820658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.820762 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.821045 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.821049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.821051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.821249 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.821341 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.821946 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.822063 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260620 21:55:46.822836 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.803430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.17.67.195:56529)
0620 21:55:46.805756 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.805769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.806231 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.806633 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.806636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.807586 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.808014 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.808034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.809661 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.809665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.810850 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.810860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.811582 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.811586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.811817 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.812432 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.812452 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.814615 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.818083 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.818087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.818089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.819112 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.819385 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.819388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.819390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.819465 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.819930 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.819933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.820013 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.821204 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.821370 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2220,"mutex_wait_us":410,"spinlock_wait_cycles":15232,"thread_start_us":209,"threads_started":1}
I20260620 21:55:46.823221 25038 heartbeater.cc:347] Connected to a master server at 127.17.67.252:46315
I20260620 21:55:46.823472 25038 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.824046 25038 heartbeater.cc:511] Master 127.17.67.252:46315 requested a full tablet report, sending...
I20260620 21:55:46.824116 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.805137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.17.67.252:46315)
0620 21:55:46.806494 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.806507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.806520 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:46.806553 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.806556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.806599 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.807331 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.808025 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.808837 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.811801 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.812969 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.815017 (+ 2048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.816582 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.816590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.816594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.816626 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.819304 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.820308 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.820311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.820314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.820400 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.822046 (+ 1646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.822049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.822051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.822291 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.823703 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.823800 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.823949 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":82,"threads_started":1}
I20260620 21:55:46.824702 25040 heartbeater.cc:347] Connected to a master server at 127.17.67.253:37903
I20260620 21:55:46.824780 25040 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.824944 25040 heartbeater.cc:511] Master 127.17.67.253:37903 requested a full tablet report, sending...
I20260620 21:55:46.825147 24580 ts_manager.cc:194] Registered new tserver with Master: c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321)
I20260620 21:55:46.825785 24394 ts_manager.cc:194] Registered new tserver with Master: c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321)
I20260620 21:55:46.826277 23793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:55:46.829327 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.805198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:46593 (local address 127.17.67.254:45653)
0620 21:55:46.805403 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.805406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.807723 (+ 2317us) server_negotiation.cc:407] Connection header received
0620 21:55:46.807769 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.807772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.807821 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.810011 (+ 2190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:46.810115 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.810996 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.812348 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.812714 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.813199 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.816282 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:46.816287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:46.816292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:46.816323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:46.819652 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.821357 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.821360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.821362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.821454 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:46.824578 (+ 3124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:46.824582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:46.824585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:46.825670 (+ 1085us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:46.825784 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.829046 (+ 3262us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.829178 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 21:55:46.829337 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.804529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:46593)
0620 21:55:46.806093 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.806107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.806208 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.808193 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.808196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.808398 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:46.808521 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.808528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.811146 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.811154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.812193 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.812202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.812859 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.812863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.812979 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.814142 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:46.814160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:46.816079 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:46.819850 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.819858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.819861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.820158 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.821546 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:46.821551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:46.821553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:46.821668 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:46.825852 (+ 4184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:46.825856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:46.828814 (+ 2958us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.828955 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.829152 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"thread_start_us":82,"threads_started":1}
W20260620 21:55:46.829444 23793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:46.829960 25039 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45653
I20260620 21:55:46.830055 25039 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:46.830252 25039 heartbeater.cc:511] Master 127.17.67.254:45653 requested a full tablet report, sending...
I20260620 21:55:46.830961 23794 ts_manager.cc:194] Re-registered known tserver with Master: c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321)
I20260620 21:55:46.831676 23794 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:46593
I20260620 21:55:46.833096 25056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: NON_VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: true } } }
I20260620 21:55:46.833228 25056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.834973 25057 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: attempting to promote NON_VOTER 0ad32c7f8dfc4150a08bcf7ed845dca5 to VOTER
I20260620 21:55:46.835325 25057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } }
I20260620 21:55:46.836517 24598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 LEARNER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:55:46.836861 24404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Refusing update from remote peer 069736e4fde948ec8c8ae9c2cdb8b16a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:55:46.836963 25057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 21:55:46.837217 25056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 21:55:46.838917 25057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.839212 24598 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.839437 24404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.840438 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.840550 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.840732 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.840852 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.840898 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.841010 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:46.841007 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.841085 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:46.841482 25065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:46.841648 25065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:46.852224 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.846388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33321 (local address 127.0.0.1:42656)
0620 21:55:46.846570 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.846586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.846669 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.847775 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.847778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.847796 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:46.847925 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.847933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849458 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.849461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.850708 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.850717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.851806 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.851891 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.851913 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.851980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.852055 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":27}
I20260620 21:55:46.852267 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.846887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.17.67.195:33321)
0620 21:55:46.847288 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.847300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.847324 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:55:46.847406 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.847411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.847612 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.848078 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:46.848212 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849367 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.850853 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.851676 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851792 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.851969 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.852024 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.852116 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":94,"threads_started":1}
I20260620 21:55:46.852551 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.846575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40621 (local address 127.0.0.1:47970)
0620 21:55:46.846813 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.846823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.846962 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.847755 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.847758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.847776 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:46.848271 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.848276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849378 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.849382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.850929 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.850937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.852044 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.852048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.852210 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.852230 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.852303 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.852400 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":91,"spinlock_wait_cycles":1920,"thread_start_us":107,"threads_started":1}
I20260620 21:55:46.853092 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.847305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:43027 (local address 127.0.0.1:51006)
0620 21:55:46.847791 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:46.847804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:46.847874 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:46.848539 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:46.848542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:46.848560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:46.848646 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.848652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849893 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.849899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.850883 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:46.850893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.852024 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.852027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:46.852780 (+ 753us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.852802 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:46.852873 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:55:46.852939 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":572,"thread_start_us":90,"threads_started":1}
I20260620 21:55:46.853140 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.847473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.17.67.193:43027)
0620 21:55:46.848090 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.848106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.848126 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:46.848199 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.848204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.848385 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.848578 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:46.848740 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849790 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851042 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.851721 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851821 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.852261 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.852909 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.853006 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":64,"threads_started":1}
I20260620 21:55:46.853989 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:46.846743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.17.67.194:40621)
0620 21:55:46.847088 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:46.847102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:46.847128 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:55:46.847212 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:46.847221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:46.847486 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:46.848061 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:46.848388 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.849261 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851050 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:46.851792 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:46.851880 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:46.851983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:46.853726 (+ 1743us) server_negotiation.cc:299] Negotiation successful
0620 21:55:46.853835 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
I20260620 21:55:46.855568 24972 tablet_service.cc:1467] Processing CreateTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b (DEFAULT_TABLE table=test-table [id=b578f62fd9a543c9ad131688f9c8ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:46.855753 24698 tablet_service.cc:1467] Processing CreateTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b (DEFAULT_TABLE table=test-table [id=b578f62fd9a543c9ad131688f9c8ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:46.856577 24972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082e9f4aede42308c0fcdb0789c5f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:46.857208 24698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082e9f4aede42308c0fcdb0789c5f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:46.857219 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b (DEFAULT_TABLE table=test-table [id=b578f62fd9a543c9ad131688f9c8ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:46.858187 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082e9f4aede42308c0fcdb0789c5f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:46.863756 25071 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Bootstrap starting.
I20260620 21:55:46.864933 25072 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Bootstrap starting.
I20260620 21:55:46.865947 25073 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Bootstrap starting.
I20260620 21:55:46.866298 25071 tablet_bootstrap.cc:654] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:46.867079 25072 tablet_bootstrap.cc:654] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:46.867210 25071 log.cc:824] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:46.867834 25072 log.cc:824] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:46.867940 25073 tablet_bootstrap.cc:654] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:46.868767 25073 log.cc:824] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:46.869730 25071 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: No bootstrap required, opened a new log
I20260620 21:55:46.869926 25071 ts_tablet_manager.cc:1397] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:55:46.870085 25072 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: No bootstrap required, opened a new log
I20260620 21:55:46.870254 25072 ts_tablet_manager.cc:1397] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:55:46.870636 25073 tablet_bootstrap.cc:492] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: No bootstrap required, opened a new log
I20260620 21:55:46.870826 25073 ts_tablet_manager.cc:1397] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:55:46.873521 25073 raft_consensus.cc:348] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.873852 25073 raft_consensus.cc:374] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:46.873909 25073 raft_consensus.cc:729] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2289f55d80d4afe89fdfcd0046df05e, State: Initialized, Role: FOLLOWER
I20260620 21:55:46.874408 25073 consensus_queue.cc:260] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.874593 25071 raft_consensus.cc:348] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.874847 25071 raft_consensus.cc:374] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:46.874915 25071 raft_consensus.cc:729] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c222d642bf5542489c06d694e543c0cb, State: Initialized, Role: FOLLOWER
I20260620 21:55:46.874593 25072 raft_consensus.cc:348] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.875018 25072 raft_consensus.cc:374] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:46.875074 25072 raft_consensus.cc:729] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdf3352f94f9a94d7d2705b4c09ff, State: Initialized, Role: FOLLOWER
I20260620 21:55:46.875440 25073 ts_tablet_manager.cc:1428] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:55:46.875617 24902 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
I20260620 21:55:46.875501 25072 consensus_queue.cc:260] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.876694 25071 consensus_queue.cc:260] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:46.876803 25072 ts_tablet_manager.cc:1428] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:46.877138 24765 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
I20260620 21:55:46.878764 25071 ts_tablet_manager.cc:1428] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 21:55:46.879108 25039 heartbeater.cc:503] Master 127.17.67.254:45653 was elected leader, sending a full tablet report...
W20260620 21:55:46.895107 24767 tablet.cc:2369] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:47.018772 25077 raft_consensus.cc:482] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:47.019049 25077 raft_consensus.cc:504] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:47.020285 25077 leader_election.cc:288] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321), 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:47.025736 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.020737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:51803 (local address 127.17.67.195:33321)
0620 21:55:47.020903 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.020910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.021116 (+ 206us) server_negotiation.cc:407] Connection header received
0620 21:55:47.021177 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.021183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.021244 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.021323 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:47.021821 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.022617 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.024359 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.024947 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.025081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.025132 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.025490 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.025615 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:55:47.025763 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.020830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:47419 (local address 127.17.67.193:43027)
0620 21:55:47.020993 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.021000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.021034 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:55:47.021101 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.021107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.021162 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.021245 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:47.021819 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.022617 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.024359 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.024979 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.025087 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.025132 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.025490 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.025650 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:55:47.025746 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.020571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33321 (local address 127.17.67.194:51803)
0620 21:55:47.021007 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:47.021029 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:47.021111 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:47.021382 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:47.021385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:47.021425 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:47.021639 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:47.021647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.022746 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.022750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:47.024218 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:47.024228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.025127 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.025131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:47.025293 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.025356 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 21:55:47.025440 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:55:47.025542 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":510,"spinlock_wait_cycles":5632}
I20260620 21:55:47.025746 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.020571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:43027 (local address 127.17.67.194:47419)
0620 21:55:47.020781 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:47.020801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:47.020948 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:47.021382 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:47.021385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:47.021427 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:47.021671 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:47.021681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.022746 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.022750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:47.024218 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:47.024228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.025127 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.025131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:47.025308 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.025351 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:55:47.025440 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:55:47.025596 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":280,"spinlock_wait_cycles":7552}
I20260620 21:55:47.026562 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0082e9f4aede42308c0fcdb0789c5f2b" candidate_uuid: "b2289f55d80d4afe89fdfcd0046df05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c222d642bf5542489c06d694e543c0cb" is_pre_election: true
I20260620 21:55:47.026576 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0082e9f4aede42308c0fcdb0789c5f2b" candidate_uuid: "b2289f55d80d4afe89fdfcd0046df05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" is_pre_election: true
I20260620 21:55:47.026871 24992 raft_consensus.cc:2454] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2289f55d80d4afe89fdfcd0046df05e in term 0.
I20260620 21:55:47.026901 24718 raft_consensus.cc:2454] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2289f55d80d4afe89fdfcd0046df05e in term 0.
I20260620 21:55:47.027601 24791 leader_election.cc:302] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2289f55d80d4afe89fdfcd0046df05e, c222d642bf5542489c06d694e543c0cb; no voters:
I20260620 21:55:47.027868 25077 raft_consensus.cc:2781] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:47.027952 25077 raft_consensus.cc:482] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:47.027993 25077 raft_consensus.cc:3037] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:47.029002 25077 raft_consensus.cc:504] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:47.029352 25077 leader_election.cc:288] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [CANDIDATE]: Term 1 election: Requested vote from peers c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321), 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:47.029656 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0082e9f4aede42308c0fcdb0789c5f2b" candidate_uuid: "b2289f55d80d4afe89fdfcd0046df05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c222d642bf5542489c06d694e543c0cb"
I20260620 21:55:47.029788 24992 raft_consensus.cc:3037] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:47.029779 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0082e9f4aede42308c0fcdb0789c5f2b" candidate_uuid: "b2289f55d80d4afe89fdfcd0046df05e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
I20260620 21:55:47.029930 24718 raft_consensus.cc:3037] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:47.030894 24992 raft_consensus.cc:2454] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2289f55d80d4afe89fdfcd0046df05e in term 1.
I20260620 21:55:47.030972 24718 raft_consensus.cc:2454] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2289f55d80d4afe89fdfcd0046df05e in term 1.
I20260620 21:55:47.031179 24791 leader_election.cc:302] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2289f55d80d4afe89fdfcd0046df05e, c222d642bf5542489c06d694e543c0cb; no voters:
I20260620 21:55:47.031354 25077 raft_consensus.cc:2781] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:47.031598 25077 raft_consensus.cc:686] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 1 LEADER]: Becoming Leader. State: Replica: b2289f55d80d4afe89fdfcd0046df05e, State: Running, Role: LEADER
I20260620 21:55:47.032163 25077 consensus_queue.cc:237] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:47.035657 23794 catalog_manager.cc:5495] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194). New cstate: current_term: 1 leader_uuid: "b2289f55d80d4afe89fdfcd0046df05e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } health_report { overall_health: UNKNOWN } } }
W20260620 21:55:47.052260 25041 tablet.cc:2369] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:55:47.084502 24904 tablet.cc:2369] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:47.088265 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.084350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.17.67.194:40621)
0620 21:55:47.084509 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.084514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.084532 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:47.084668 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.084672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.084743 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.084870 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:47.085152 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.085808 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.086763 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.087079 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.087176 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.087438 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:47.087875 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:47.087969 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.088018 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.088070 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:55:47.094323 24718 raft_consensus.cc:1261] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Refusing update from remote peer b2289f55d80d4afe89fdfcd0046df05e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:47.094323 24992 raft_consensus.cc:1261] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Refusing update from remote peer b2289f55d80d4afe89fdfcd0046df05e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:47.095191 25077 consensus_queue.cc:1035] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:47.095438 25080 consensus_queue.cc:1035] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:47.107203 25084 mvcc.cc:205] Tried to move back new op lower bound from 7299041472890101760 to 7299041472646471680. Current Snapshot: MvccSnapshot[applied={T|T < 7299041472890101760 or (T in {7299041472890101760})}]
I20260620 21:55:47.116065 23794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:47.116268 23794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:47.120988 23794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 069736e4fde948ec8c8ae9c2cdb8b16a: Sending DeleteTablet for 3 replicas of tablet 0082e9f4aede42308c0fcdb0789c5f2b
I20260620 21:55:47.121754 24972 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:47 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:42656
I20260620 21:55:47.121877 24835 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:47 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:47970
I20260620 21:55:47.121992 24698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082e9f4aede42308c0fcdb0789c5f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:47 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:51006
I20260620 21:55:47.122439 25094 tablet_replica.cc:330] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: stopping tablet replica
I20260620 21:55:47.122439 25093 tablet_replica.cc:330] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: stopping tablet replica
I20260620 21:55:47.122757 25094 raft_consensus.cc:2229] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:47.122757 25093 raft_consensus.cc:2229] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:47.123126 25093 raft_consensus.cc:2258] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:47.123167 25094 raft_consensus.cc:2258] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:47.124718 25093 ts_tablet_manager.cc:1905] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:47.126550 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 23762
I20260620 21:55:47.128170 25093 ts_tablet_manager.cc:1918] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:47.128271 25093 log.cc:1196] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/0082e9f4aede42308c0fcdb0789c5f2b
I20260620 21:55:47.128595 25093 ts_tablet_manager.cc:1939] T 0082e9f4aede42308c0fcdb0789c5f2b P b2289f55d80d4afe89fdfcd0046df05e: Deleting consensus metadata
I20260620 21:55:47.132917 25094 ts_tablet_manager.cc:1905] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:47.134879 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.124130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.17.67.253:37903)
0620 21:55:47.124236 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.124240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.124253 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:47.124298 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.124301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.124344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.124419 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:47.125948 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.126718 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.128250 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.128655 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.128755 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.131043 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:47.131047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:47.131050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:47.131075 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:47.132975 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.133457 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.133461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.133462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.133549 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.133892 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.133895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.133897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.134098 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:47.134200 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.134582 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.134717 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 21:55:47.135627 24790 connection.cc:535] server connection from 127.0.0.1:47970 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:55:47.136234 25094 ts_tablet_manager.cc:1918] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:47.136417 25094 log.cc:1196] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/0082e9f4aede42308c0fcdb0789c5f2b
I20260620 21:55:47.136817 25094 ts_tablet_manager.cc:1939] T 0082e9f4aede42308c0fcdb0789c5f2b P 8f6fdf3352f94f9a94d7d2705b4c09ff: Deleting consensus metadata
I20260620 21:55:47.137197 25092 tablet_replica.cc:330] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: stopping tablet replica
I20260620 21:55:47.137535 25092 raft_consensus.cc:2229] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:47.137887 25092 raft_consensus.cc:2258] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:55:47.138346 24654 connection.cc:441] server connection from 127.0.0.1:51006 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51006 (request call id 1) could send its response
I20260620 21:55:47.141680 25092 ts_tablet_manager.cc:1905] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:47.144485 25092 ts_tablet_manager.cc:1918] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:47.144593 25092 log.cc:1196] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/0082e9f4aede42308c0fcdb0789c5f2b
I20260620 21:55:47.144937 25092 ts_tablet_manager.cc:1939] T 0082e9f4aede42308c0fcdb0789c5f2b P c222d642bf5542489c06d694e543c0cb: Deleting consensus metadata
W20260620 21:55:47.146117 24927 connection.cc:441] server connection from 127.0.0.1:42656 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42656 (request call id 1) could send its response
I20260620 21:55:47.146237 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.123875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.17.67.252:46315)
0620 21:55:47.124006 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.124011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.136243 (+ 12232us) server_negotiation.cc:407] Connection header received
0620 21:55:47.137271 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.137274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.137333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.137385 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:47.138286 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.138996 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.139782 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.140058 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.140155 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.142143 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:47.142147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:47.142150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:47.142176 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:47.143887 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.144364 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.144368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.144370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.144453 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.144874 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.144879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.144880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.145401 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:47.145530 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.145920 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.146049 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:47.150933 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.140245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.17.67.253:37903)
0620 21:55:47.140361 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.140365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.140674 (+ 309us) server_negotiation.cc:407] Connection header received
0620 21:55:47.140862 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.140865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.140915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.140969 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:47.141933 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.142709 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.144070 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.144432 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.144513 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.146900 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:47.146904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:47.146908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:47.146964 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:47.148976 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.149467 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.149471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.149473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.149560 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.149923 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.149927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.149929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.150155 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:47.150255 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.150629 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.150782 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:47.162371 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:47.154247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.17.67.252:46315)
0620 21:55:47.154382 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:47.154385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:47.154430 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:55:47.154556 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:47.154558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:47.154591 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:47.154655 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:47.155536 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.156059 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.157244 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:47.157468 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:47.157555 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:47.159209 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:47.159212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:47.159214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:47.159236 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:47.160686 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.161169 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.161171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.161172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.161242 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:47.161513 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:47.161515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:47.161516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:47.161785 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:47.161885 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:47.162133 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 21:55:47.162221 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:55:48.109458 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.109052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:40455)
0620 21:55:48.109286 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.109368 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
W20260620 21:55:48.109797 24765 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:45653 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.112021 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.111473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:41645)
0620 21:55:48.111746 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.111865 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
W20260620 21:55:48.112360 25039 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:45653 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.116055 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.115581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:45205)
0620 21:55:48.115841 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.115952 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
W20260620 21:55:48.116326 24902 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:45653 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.621253 25104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 069736e4fde948ec8c8ae9c2cdb8b16a)
I20260620 21:55:48.621486 25104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } }
I20260620 21:55:48.622865 25104 leader_election.cc:288] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653), 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252:46315)
I20260620 21:55:48.623453 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.623003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50728)
0620 21:55:48.623250 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.623331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
W20260620 21:55:48.623791 24380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.624758 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.624502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50744)
0620 21:55:48.624638 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.624693 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 21:55:48.624929 24380 leader_election.cc:334] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653): Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.624974 25109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 069736e4fde948ec8c8ae9c2cdb8b16a)
I20260620 21:55:48.625152 25109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } }
I20260620 21:55:48.626170 25109 leader_election.cc:288] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653), 063c27a815f9403db7b66be7e7099f15 (127.17.67.253:37903)
I20260620 21:55:48.626719 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.626312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50750)
0620 21:55:48.626544 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.626620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
W20260620 21:55:48.626892 24570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.627772 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.627470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50762)
0620 21:55:48.627611 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.627707 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 21:55:48.627894 24570 leader_election.cc:334] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653): Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.627944 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.623359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46315 (local address 127.0.0.1:42216)
0620 21:55:48.623607 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.623623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.623727 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.624038 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.624040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.624054 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.624143 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.624148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.625171 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.625175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.626511 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.626523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.627474 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.627478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.627611 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.627646 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.627728 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.627812 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":25,"thread_start_us":98,"threads_started":1}
I20260620 21:55:48.628026 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.623538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.17.67.252:46315)
0620 21:55:48.623794 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.623801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.623819 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:48.623872 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.623877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.623927 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.624028 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.624259 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.625019 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.626662 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.627369 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.627486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.627557 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.627761 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.627846 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260620 21:55:48.628405 24598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "063c27a815f9403db7b66be7e7099f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" is_pre_election: true
I20260620 21:55:48.628605 24598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063c27a815f9403db7b66be7e7099f15 in term 1.
I20260620 21:55:48.628933 24380 leader_election.cc:302] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 063c27a815f9403db7b66be7e7099f15, 0ad32c7f8dfc4150a08bcf7ed845dca5; no voters: 069736e4fde948ec8c8ae9c2cdb8b16a
I20260620 21:55:48.629165 25104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:55:48.629242 25104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 069736e4fde948ec8c8ae9c2cdb8b16a)
I20260620 21:55:48.629300 25104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:55:48.630653 25104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } }
I20260620 21:55:48.631044 25104 leader_election.cc:288] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 election: Requested vote from peers 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653), 0ad32c7f8dfc4150a08bcf7ed845dca5 (127.17.67.252:46315)
I20260620 21:55:48.631445 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.631125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50770)
0620 21:55:48.631329 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.631382 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 21:55:48.631469 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.626322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:37903 (local address 127.0.0.1:41160)
0620 21:55:48.626595 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.626607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.626710 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.627306 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.627310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.627326 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.627408 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.627417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.628876 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.628879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.629936 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.629945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.631016 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.631020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.631115 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.631143 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.631248 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.631322 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":34,"thread_start_us":66,"threads_started":1}
I20260620 21:55:48.631520 24598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "063c27a815f9403db7b66be7e7099f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5"
I20260620 21:55:48.631650 24598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:55:48.631846 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.626597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.17.67.253:37903)
0620 21:55:48.627015 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.627024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.627038 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:48.627077 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.627079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.627120 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.627202 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.627534 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.628742 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.630054 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.630780 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.630892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.630952 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.631669 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.631749 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":315,"thread_start_us":106,"threads_started":1}
I20260620 21:55:48.632215 24404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "063c27a815f9403db7b66be7e7099f15" is_pre_election: true
I20260620 21:55:48.632447 24404 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ad32c7f8dfc4150a08bcf7ed845dca5 in current term 2: Already voted for candidate 063c27a815f9403db7b66be7e7099f15 in this term.
I20260620 21:55:48.632835 24572 leader_election.cc:302] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ad32c7f8dfc4150a08bcf7ed845dca5; no voters: 063c27a815f9403db7b66be7e7099f15, 069736e4fde948ec8c8ae9c2cdb8b16a
I20260620 21:55:48.632872 24598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063c27a815f9403db7b66be7e7099f15 in term 2.
I20260620 21:55:48.633035 25109 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:55:48.633266 24380 leader_election.cc:302] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063c27a815f9403db7b66be7e7099f15, 0ad32c7f8dfc4150a08bcf7ed845dca5; no voters:
I20260620 21:55:48.633411 25104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:55:48.633674 25104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [term 2 LEADER]: Becoming Leader. State: Replica: 063c27a815f9403db7b66be7e7099f15, State: Running, Role: LEADER
I20260620 21:55:48.633788 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.633521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50780)
0620 21:55:48.633675 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.633733 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 21:55:48.633882 24380 leader_election.cc:334] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653): Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
I20260620 21:55:48.633953 25104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } }
I20260620 21:55:48.634925 25114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063c27a815f9403db7b66be7e7099f15. Latest consensus state: current_term: 2 leader_uuid: "063c27a815f9403db7b66be7e7099f15" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:48.635042 25114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:48.635411 25116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:48.637817 25116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:48.638245 25116 catalog_manager.cc:1259] Loaded cluster ID: bd6ad43369024de2807a0ac9307a9ccf
I20260620 21:55:48.638319 25116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:48.638926 25116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:48.639335 25116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Loaded TSK: 0
I20260620 21:55:48.640020 25116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:48.682264 24395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:48.685433 24395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:48.688949 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.688675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50808)
0620 21:55:48.688805 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.688870 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:55:48.688944 24598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [term 2 FOLLOWER]: Refusing update from remote peer 063c27a815f9403db7b66be7e7099f15: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 21:55:48.689697 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.689414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.0.0.1:50818)
0620 21:55:48.689522 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.689637 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 21:55:48.689667 25114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 21:55:48.689839 24380 consensus_peers.cc:487] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 -> Peer 069736e4fde948ec8c8ae9c2cdb8b16a (127.17.67.254:45653): Couldn't send request to peer 069736e4fde948ec8c8ae9c2cdb8b16a. Status: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:55:48.691161 25109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063c27a815f9403db7b66be7e7099f15. Latest consensus state: current_term: 2 leader_uuid: "063c27a815f9403db7b66be7e7099f15" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:48.691283 25109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:48.692502 25104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "063c27a815f9403db7b66be7e7099f15" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:48.692644 25104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:48.692778 25109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "063c27a815f9403db7b66be7e7099f15" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "069736e4fde948ec8c8ae9c2cdb8b16a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45653 } } peers { permanent_uuid: "063c27a815f9403db7b66be7e7099f15" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "0ad32c7f8dfc4150a08bcf7ed845dca5" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46315 } attrs { promote: false } } }
I20260620 21:55:48.692896 25109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad32c7f8dfc4150a08bcf7ed845dca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:48.703248 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.698407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40621 (local address 127.0.0.1:47978)
0620 21:55:48.698544 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.698560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.698651 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.699296 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.699301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.699323 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.699449 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.699456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700666 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.700669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.701927 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.701937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.702787 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.702790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.702892 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.702935 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.702987 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.703077 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":32}
I20260620 21:55:48.703258 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.698920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.17.67.195:33321)
0620 21:55:48.699484 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.699489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.699503 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:48.699554 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.699557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.699604 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.700048 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.700179 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700821 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.701944 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.702589 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.702896 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.702967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.703052 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.703134 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":59,"threads_started":1}
I20260620 21:55:48.703289 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.698691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33321 (local address 127.0.0.1:42664)
0620 21:55:48.698954 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.698967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.699040 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.699730 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.699733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.699752 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.699885 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.699892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700953 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.700958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.701824 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.701832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.702734 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.702740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.702933 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.702962 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.703041 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.703126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":62}
I20260620 21:55:48.703591 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.699002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:43027 (local address 127.0.0.1:51016)
0620 21:55:48.699271 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.699286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.699353 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.699812 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.699815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.699831 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.699957 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.699963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700997 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.701001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.702351 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.702361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.703242 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.703245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.703334 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.703362 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.703415 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.703486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260620 21:55:48.703846 24972 tablet_service.cc:1467] Processing CreateTablet for tablet 0517a6486c3f4c79bda5f63554891779 (DEFAULT_TABLE table=test-table [id=2f9d7c70ea1b49ef952298a12b8986e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:48.704147 24972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517a6486c3f4c79bda5f63554891779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:48.704195 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.698595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.17.67.194:40621)
0620 21:55:48.699010 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.699015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.699031 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:55:48.699083 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.699088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.699163 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.699554 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.699674 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700469 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.702046 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.702677 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.703865 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.703930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.703968 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.704055 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":74,"threads_started":1}
I20260620 21:55:48.704262 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.699340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.17.67.193:43027)
0620 21:55:48.699577 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.699582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.699596 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:48.699647 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.699650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.699695 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.699882 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.700040 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.700881 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.702484 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.703119 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.703902 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.703992 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.704029 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.704129 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260620 21:55:48.704641 24835 tablet_service.cc:1467] Processing CreateTablet for tablet 0517a6486c3f4c79bda5f63554891779 (DEFAULT_TABLE table=test-table [id=2f9d7c70ea1b49ef952298a12b8986e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:48.704887 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517a6486c3f4c79bda5f63554891779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:48.704833 24698 tablet_service.cc:1467] Processing CreateTablet for tablet 0517a6486c3f4c79bda5f63554891779 (DEFAULT_TABLE table=test-table [id=2f9d7c70ea1b49ef952298a12b8986e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:48.705060 24698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0517a6486c3f4c79bda5f63554891779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:48.706254 25127 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Bootstrap starting.
I20260620 21:55:48.707077 25127 tablet_bootstrap.cc:654] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:48.707456 25128 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff: Bootstrap starting.
I20260620 21:55:48.708194 25128 tablet_bootstrap.cc:654] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:48.708364 25127 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: No bootstrap required, opened a new log
I20260620 21:55:48.708449 25127 ts_tablet_manager.cc:1397] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:48.708861 25129 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Bootstrap starting.
I20260620 21:55:48.708917 25127 raft_consensus.cc:348] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.709020 25127 raft_consensus.cc:374] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:48.709056 25127 raft_consensus.cc:729] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c222d642bf5542489c06d694e543c0cb, State: Initialized, Role: FOLLOWER
I20260620 21:55:48.709190 25127 consensus_queue.cc:260] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.709498 25128 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff: No bootstrap required, opened a new log
I20260620 21:55:48.709530 25127 ts_tablet_manager.cc:1428] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:48.709602 25128 ts_tablet_manager.cc:1397] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:48.710105 25128 raft_consensus.cc:348] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.710220 25128 raft_consensus.cc:374] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:48.710254 25128 raft_consensus.cc:729] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdf3352f94f9a94d7d2705b4c09ff, State: Initialized, Role: FOLLOWER
I20260620 21:55:48.710340 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.709907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:44655)
0620 21:55:48.710179 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.710239 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260620 21:55:48.710870 25128 consensus_queue.cc:260] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.711230 25128 ts_tablet_manager.cc:1428] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:55:48.712672 25129 tablet_bootstrap.cc:654] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:48.712651 25040 heartbeater.cc:503] Master 127.17.67.253:37903 was elected leader, sending a full tablet report...
I20260620 21:55:48.713438 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.711593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:41943)
0620 21:55:48.713301 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.713359 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1632,"thread_start_us":95,"threads_started":1}
I20260620 21:55:48.713976 25129 tablet_bootstrap.cc:492] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: No bootstrap required, opened a new log
I20260620 21:55:48.714063 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.710782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:34957)
0620 21:55:48.713936 (+ 3154us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.713996 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3080}
I20260620 21:55:48.714068 25129 ts_tablet_manager.cc:1397] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 21:55:48.714455 25129 raft_consensus.cc:348] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.714588 25129 raft_consensus.cc:374] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:48.714628 25129 raft_consensus.cc:729] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2289f55d80d4afe89fdfcd0046df05e, State: Initialized, Role: FOLLOWER
I20260620 21:55:48.714766 25129 consensus_queue.cc:260] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.715116 25129 ts_tablet_manager.cc:1428] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:48.715128 24766 heartbeater.cc:503] Master 127.17.67.253:37903 was elected leader, sending a full tablet report...
I20260620 21:55:48.716287 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.716079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:59869)
0620 21:55:48.716172 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.716226 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 21:55:48.716719 24903 heartbeater.cc:503] Master 127.17.67.253:37903 was elected leader, sending a full tablet report...
I20260620 21:55:48.717338 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.715361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:47203)
0620 21:55:48.717187 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.717245 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1754,"thread_start_us":92,"threads_started":1}
I20260620 21:55:48.717962 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.717741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:44613)
0620 21:55:48.717871 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.717914 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 21:55:48.722493 25132 raft_consensus.cc:482] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:48.722617 25132 raft_consensus.cc:504] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.723654 25132 leader_election.cc:288] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621), 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:48.726954 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.723828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:58469 (local address 127.17.67.194:40621)
0620 21:55:48.723930 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.723934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.723947 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:48.724038 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.724044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.724101 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.724164 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.724420 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.724922 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726035 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.726437 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726557 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.726606 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.726786 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.726853 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:55:48.726954 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.723757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40621 (local address 127.17.67.195:58469)
0620 21:55:48.723839 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.723855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.723947 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.724187 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.724190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.724207 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.724318 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.724323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.725038 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.725042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.725910 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.725919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.726552 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.726668 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.726686 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.726773 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.726843 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 21:55:48.727289 25134 raft_consensus.cc:482] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:48.727406 25134 raft_consensus.cc:504] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.727453 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "c222d642bf5542489c06d694e543c0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2289f55d80d4afe89fdfcd0046df05e" is_pre_election: true
I20260620 21:55:48.727563 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.723881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:43027 (local address 127.17.67.195:41287)
0620 21:55:48.724156 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.724171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.724265 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.724494 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.724497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.724512 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.724620 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.724625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.725319 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.725324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.726214 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.726224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.726852 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.726948 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.726975 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.727331 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.727419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260620 21:55:48.727648 24855 raft_consensus.cc:2454] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c222d642bf5542489c06d694e543c0cb in term 0.
I20260620 21:55:48.727311 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.724193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:41287 (local address 127.17.67.193:43027)
0620 21:55:48.724299 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.724304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.724319 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:48.724360 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.724363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.724398 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.724487 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.724726 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.725211 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726331 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.726731 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.726858 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.726903 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.727133 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.727209 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:55:48.728014 24927 leader_election.cc:302] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2289f55d80d4afe89fdfcd0046df05e, c222d642bf5542489c06d694e543c0cb; no voters:
I20260620 21:55:48.728173 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "c222d642bf5542489c06d694e543c0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" is_pre_election: true
I20260620 21:55:48.728288 25132 raft_consensus.cc:2781] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:48.728308 24718 raft_consensus.cc:2454] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c222d642bf5542489c06d694e543c0cb in term 0.
I20260620 21:55:48.728349 25132 raft_consensus.cc:482] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:48.728391 25132 raft_consensus.cc:3037] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:48.728580 25134 leader_election.cc:288] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621), c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321)
I20260620 21:55:48.729703 25132 raft_consensus.cc:504] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.730052 25132 leader_election.cc:288] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [CANDIDATE]: Term 1 election: Requested vote from peers b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621), 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027)
I20260620 21:55:48.730352 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "c222d642bf5542489c06d694e543c0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2289f55d80d4afe89fdfcd0046df05e"
I20260620 21:55:48.730423 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "c222d642bf5542489c06d694e543c0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff"
I20260620 21:55:48.730460 24855 raft_consensus.cc:3037] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:48.730520 24718 raft_consensus.cc:3037] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:48.731621 24855 raft_consensus.cc:2454] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c222d642bf5542489c06d694e543c0cb in term 1.
I20260620 21:55:48.731684 24718 raft_consensus.cc:2454] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c222d642bf5542489c06d694e543c0cb in term 1.
I20260620 21:55:48.731995 24925 leader_election.cc:302] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8f6fdf3352f94f9a94d7d2705b4c09ff, b2289f55d80d4afe89fdfcd0046df05e, c222d642bf5542489c06d694e543c0cb; no voters:
I20260620 21:55:48.732246 25132 raft_consensus.cc:2781] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:48.732494 25132 raft_consensus.cc:686] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 1 LEADER]: Becoming Leader. State: Replica: c222d642bf5542489c06d694e543c0cb, State: Running, Role: LEADER
I20260620 21:55:48.732746 25132 consensus_queue.cc:237] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } }
I20260620 21:55:48.732944 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.728672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40621 (local address 127.17.67.193:33375)
0620 21:55:48.728881 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.728895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.728981 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.729273 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.729275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.729292 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.729375 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.729381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.730315 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.730318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.731585 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.731598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.732527 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.732530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.732649 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.732671 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.732741 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.732814 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 21:55:48.733364 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.728856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:33375 (local address 127.17.67.194:40621)
0620 21:55:48.728959 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.728963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.728975 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:55:48.729070 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.729073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.729113 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.729180 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.729499 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.730199 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.731730 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.732309 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.732415 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.732459 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.733191 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.733260 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 21:55:48.733385 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.728830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33321 (local address 127.17.67.193:55125)
0620 21:55:48.729259 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.729273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:48.729399 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:48.729772 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:48.729775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:48.729792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:48.729895 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.729901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.730855 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.730858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.732018 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:48.732028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.732964 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.732967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:48.733097 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.733133 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:55:48.733202 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:55:48.733259 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":90,"threads_started":1}
I20260620 21:55:48.733709 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.729109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:55125 (local address 127.17.67.195:33321)
0620 21:55:48.729221 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.729225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.729340 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:55:48.729507 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.729512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.729587 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.729639 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:48.730017 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.730717 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.732210 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.732771 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.732866 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.732908 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.733508 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.733612 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:55:48.733667 24855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2289f55d80d4afe89fdfcd0046df05e" is_pre_election: true
I20260620 21:55:48.733837 24855 raft_consensus.cc:2379] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f6fdf3352f94f9a94d7d2705b4c09ff in current term 1: Already voted for candidate c222d642bf5542489c06d694e543c0cb in this term.
I20260620 21:55:48.734048 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.733132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:43753)
0620 21:55:48.733922 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.733990 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726}
I20260620 21:55:48.734622 24992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0517a6486c3f4c79bda5f63554891779" candidate_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c222d642bf5542489c06d694e543c0cb" is_pre_election: true
I20260620 21:55:48.735023 24654 leader_election.cc:302] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdf3352f94f9a94d7d2705b4c09ff; no voters: b2289f55d80d4afe89fdfcd0046df05e, c222d642bf5542489c06d694e543c0cb
I20260620 21:55:48.735291 25134 raft_consensus.cc:2726] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 21:55:48.735188 24395 catalog_manager.cc:5495] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c222d642bf5542489c06d694e543c0cb (127.17.67.195). New cstate: current_term: 1 leader_uuid: "c222d642bf5542489c06d694e543c0cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c222d642bf5542489c06d694e543c0cb" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:48.757061 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.753458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.17.67.195:33321)
0620 21:55:48.753614 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:48.753618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:48.753633 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:48.753801 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:48.753804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:48.753859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:48.753980 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:48.754215 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.754735 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.755615 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:48.755840 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:48.755954 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:48.756220 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:48.756670 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:48.756802 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:48.756863 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 21:55:48.756916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:48.760146 24718 raft_consensus.cc:1261] T 0517a6486c3f4c79bda5f63554891779 P 8f6fdf3352f94f9a94d7d2705b4c09ff [term 1 FOLLOWER]: Refusing update from remote peer c222d642bf5542489c06d694e543c0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:48.760180 24855 raft_consensus.cc:1261] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Refusing update from remote peer c222d642bf5542489c06d694e543c0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:48.760821 25141 consensus_queue.cc:1035] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2289f55d80d4afe89fdfcd0046df05e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:48.761027 25132 consensus_queue.cc:1035] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6fdf3352f94f9a94d7d2705b4c09ff" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:48.762791 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.762320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:46353)
0620 21:55:48.762635 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.762708 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260620 21:55:48.764307 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.762698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:41573)
0620 21:55:48.764178 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.764237 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1404}
I20260620 21:55:48.767550 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.767300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:47475)
0620 21:55:48.767407 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.767477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 21:55:48.770931 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.770544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.195:40535)
0620 21:55:48.770793 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.770853 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 21:55:48.771492 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.770144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.194:44399)
0620 21:55:48.771357 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.771410 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1157,"thread_start_us":99,"threads_started":1}
I20260620 21:55:48.771670 25147 mvcc.cc:205] Tried to move back new op lower bound from 7299041479715168256 to 7299041479611953152. Current Snapshot: MvccSnapshot[applied={T|T < 7299041479715168256}]
I20260620 21:55:48.772197 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:48.771870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45653 (local address 127.17.67.193:56907)
0620 21:55:48.772058 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.772115 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:45653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 21:55:48.777660 24394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:48.777820 24394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:48.781952 24394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 063c27a815f9403db7b66be7e7099f15: Sending DeleteTablet for 3 replicas of tablet 0517a6486c3f4c79bda5f63554891779
I20260620 21:55:48.782915 24972 tablet_service.cc:1514] Processing DeleteTablet for tablet 0517a6486c3f4c79bda5f63554891779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:48 UTC) from {username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:42664
I20260620 21:55:48.782976 24835 tablet_service.cc:1514] Processing DeleteTablet for tablet 0517a6486c3f4c79bda5f63554891779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:48 UTC) from {username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:47978
I20260620 21:55:48.782980 24698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0517a6486c3f4c79bda5f63554891779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:48 UTC) from {username='oryx', principal='oryx/127.17.67.253@KRBTEST.COM'} at 127.0.0.1:51016
I20260620 21:55:48.783455 25155 tablet_replica.cc:330] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: stopping tablet replica
I20260620 21:55:48.783592 25155 raft_consensus.cc:2229] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:48.783906 25155 raft_consensus.cc:2258] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:48.784338 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24539
I20260620 21:55:48.785858 25155 ts_tablet_manager.cc:1905] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:48.788601 25155 ts_tablet_manager.cc:1918] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:48.788688 25155 log.cc:1196] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/0517a6486c3f4c79bda5f63554891779
I20260620 21:55:48.788964 25155 ts_tablet_manager.cc:1939] T 0517a6486c3f4c79bda5f63554891779 P c222d642bf5542489c06d694e543c0cb: Deleting consensus metadata
I20260620 21:55:48.789682 24382 catalog_manager.cc:4841] TS c222d642bf5542489c06d694e543c0cb (127.17.67.195:33321): tablet 0517a6486c3f4c79bda5f63554891779 (table test-table [id=2f9d7c70ea1b49ef952298a12b8986e5]) successfully deleted
I20260620 21:55:48.793799 25157 tablet_replica.cc:330] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: stopping tablet replica
I20260620 21:55:48.793941 25157 raft_consensus.cc:2229] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:48.794118 25157 raft_consensus.cc:2258] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:48.794826 25157 ts_tablet_manager.cc:1905] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:48.796864 25157 ts_tablet_manager.cc:1918] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:48.796957 25157 log.cc:1196] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/0517a6486c3f4c79bda5f63554891779
I20260620 21:55:48.797264 25157 ts_tablet_manager.cc:1939] T 0517a6486c3f4c79bda5f63554891779 P b2289f55d80d4afe89fdfcd0046df05e: Deleting consensus metadata
I20260620 21:55:48.797967 24381 catalog_manager.cc:4841] TS b2289f55d80d4afe89fdfcd0046df05e (127.17.67.194:40621): tablet 0517a6486c3f4c79bda5f63554891779 (table test-table [id=2f9d7c70ea1b49ef952298a12b8986e5]) successfully deleted
W20260620 21:55:48.799275 25105 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:48.798920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:43027 (local address 127.0.0.1:51018)
0620 21:55:48.799096 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:48.799143 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:43027: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 21:55:48.799297 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24772
W20260620 21:55:48.800083 24379 catalog_manager.cc:4584] TS 8f6fdf3352f94f9a94d7d2705b4c09ff (127.17.67.193:43027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0517a6486c3f4c79bda5f63554891779: Network error: Client connection negotiation failed: client connection to 127.17.67.193:43027: connect: Connection reset by peer (error 104)
I20260620 21:55:48.807163 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24908
I20260620 21:55:48.812136 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24258
I20260620 21:55:48.818132 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 24361
2026-06-20T21:55:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5455 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:48 dist-test-slave-cx8d krb5kdc[25160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:48 dist-test-slave-cx8d krb5kdc[25160](info): set up 2 sockets
Jun 20 21:55:48 dist-test-slave-cx8d krb5kdc[25160](info): commencing operation
krb5kdc: starting...
W20260620 21:55:49.871765 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:49 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab.
Entry for principal oryx/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab.
I20260620 21:55:50.000443 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39359
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab
--principal=oryx/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:50.109265 25173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:50.109524 25173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:50.109623 25173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:50.113535 25173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:50.113601 25173 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:55:50.113619 25173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:50.113637 25173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:50.113653 25173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:50.113670 25173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:50.118925 25173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.254.keytab
--principal=oryx/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:50.119877 25173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:50.120816 25173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:50.129014 25179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.129082 25178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.129170 25173 server_base.cc:1034] running on GCE node
W20260620 21:55:50.129015 25181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.130005 25173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:50.131157 25173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:50.132362 25173 hybrid_clock.cc:648] HybridClock initialized: now 1781992550132349 us; error 25 us; skew 500 ppm
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:50.135234 25173 init.cc:377] Logged in from keytab as oryx/127.17.67.254@KRBTEST.COM (short username oryx)
I20260620 21:55:50.136052 25173 webserver.cc:460] Webserver started at http://127.17.67.254:40771/ using document root <none> and password file <none>
I20260620 21:55:50.136544 25173 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:50.136608 25173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:50.136765 25173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:50.138777 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "546520af25be423f8296d1b4ebf3483e"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "927b4793f94d9c3289098a39931edfc3"
server_key_iv: "ca495551fe7097c300b252629f668898"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.139283 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "546520af25be423f8296d1b4ebf3483e"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "927b4793f94d9c3289098a39931edfc3"
server_key_iv: "ca495551fe7097c300b252629f668898"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.142966 25173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:55:50.145195 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.146504 25173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:55:50.146644 25173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "546520af25be423f8296d1b4ebf3483e"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "927b4793f94d9c3289098a39931edfc3"
server_key_iv: "ca495551fe7097c300b252629f668898"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.146786 25173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:50.169723 25173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:50.173091 25173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:50.173300 25173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:50.182324 25173 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39359
I20260620 21:55:50.182337 25250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39359 every 8 connection(s)
I20260620 21:55:50.183411 25173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:50.186241 25251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:50.186765 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25173
I20260620 21:55:50.186925 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:50.187274 17679 external_mini_cluster.cc:1428] Setting key b8516db9d367b618a323a013b934f5e9
I20260620 21:55:50.192085 25251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Bootstrap starting.
I20260620 21:55:50.194319 25251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:50.195044 25251 log.cc:824] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:50.196914 25251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: No bootstrap required, opened a new log
I20260620 21:55:50.200187 25251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } }
I20260620 21:55:50.200452 25251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:50.200541 25251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546520af25be423f8296d1b4ebf3483e, State: Initialized, Role: FOLLOWER
I20260620 21:55:50.201035 25251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } }
I20260620 21:55:50.201191 25251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:50.201256 25251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:50.201360 25251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:50.202373 25251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } }
I20260620 21:55:50.202700 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.188706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.17.67.254:39359)
0620 21:55:50.189122 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.189137 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.189195 (+ 58us) server_negotiation.cc:407] Connection header received
0620 21:55:50.189920 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.189938 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.190285 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.190760 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.191608 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.192709 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.193648 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.194160 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.194354 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.197095 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.197100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.197113 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.197139 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.199692 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.200177 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.200190 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.200195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.200319 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.200626 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.200632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.200633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.201036 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.201282 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.201539 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.201884 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260620 21:55:50.202715 25251 leader_election.cc:302] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 546520af25be423f8296d1b4ebf3483e; no voters:
I20260620 21:55:50.203125 25251 leader_election.cc:288] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:50.203204 25256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:50.203385 25256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [term 1 LEADER]: Becoming Leader. State: Replica: 546520af25be423f8296d1b4ebf3483e, State: Running, Role: LEADER
I20260620 21:55:50.203661 25256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } }
I20260620 21:55:50.204406 25251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:50.205329 25258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "546520af25be423f8296d1b4ebf3483e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } } }
I20260620 21:55:50.205469 25258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:50.205711 25257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 546520af25be423f8296d1b4ebf3483e. Latest consensus state: current_term: 1 leader_uuid: "546520af25be423f8296d1b4ebf3483e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546520af25be423f8296d1b4ebf3483e" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39359 } } }
I20260620 21:55:50.205837 25257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:50.205909 25265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:50.208990 25265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:50.214622 25265 catalog_manager.cc:1347] Generated new cluster ID: e1dd8166172c41aaa6f15beb9150e9cf
I20260620 21:55:50.214726 25265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:50.229830 25265 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:50.230871 25265 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:50.239073 25265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Generated new TSK 0
I20260620 21:55:50.239838 25265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
I20260620 21:55:50.272784 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39359
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:50.385274 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:50.385555 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:50.385625 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:50.389724 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:50.389787 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:50.389827 25277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:50.389860 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:50.395149 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:50.396159 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:50.397125 25277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:50.406647 25285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.406656 25282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.406656 25283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.407009 25277 server_base.cc:1034] running on GCE node
I20260620 21:55:50.407357 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:50.407965 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:50.409149 25277 hybrid_clock.cc:648] HybridClock initialized: now 1781992550409126 us; error 33 us; skew 500 ppm
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:50.412021 25277 init.cc:377] Logged in from keytab as oryx/127.17.67.193@KRBTEST.COM (short username oryx)
I20260620 21:55:50.412879 25277 webserver.cc:460] Webserver started at http://127.17.67.193:36043/ using document root <none> and password file <none>
I20260620 21:55:50.413396 25277 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:50.413455 25277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:50.413689 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:50.415648 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "f4f2240aef90489bb774464c943b8237"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "b97fee670cfd670224357b48d5b7d5ea"
server_key_iv: "5a2597a6e1dafd2e1da8f9728174338e"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.416172 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "f4f2240aef90489bb774464c943b8237"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "b97fee670cfd670224357b48d5b7d5ea"
server_key_iv: "5a2597a6e1dafd2e1da8f9728174338e"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.419863 25277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:55:50.422199 25292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.423425 25277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:50.423570 25277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "f4f2240aef90489bb774464c943b8237"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "b97fee670cfd670224357b48d5b7d5ea"
server_key_iv: "5a2597a6e1dafd2e1da8f9728174338e"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.423681 25277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:50.440733 25277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:50.444123 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:50.444334 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:50.445843 25277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:50.445904 25277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.445953 25277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:50.445979 25277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:50.456703 25277 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:46035
I20260620 21:55:50.456841 25407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:46035 every 8 connection(s)
I20260620 21:55:50.457746 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:50.459141 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25277
I20260620 21:55:50.459299 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:50.459635 17679 external_mini_cluster.cc:1428] Setting key 9355c44d26d74d280e1f5162ff9dffc0
I20260620 21:55:50.460317 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.446903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:42943 (local address 127.17.67.254:39359)
0620 21:55:50.447051 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.447056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.447642 (+ 586us) server_negotiation.cc:407] Connection header received
0620 21:55:50.448810 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.448813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.448875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.448958 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.450520 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.451473 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.452971 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.453226 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.453338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.456278 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.456282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.456284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.456312 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.457979 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.458548 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.458553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.458555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.458648 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.459016 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.459021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.459023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.459214 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.459330 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.459948 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.460149 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:55:50.461194 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.447014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.193:42943)
0620 21:55:50.447478 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.447526 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.448558 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.449127 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.449137 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.449695 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.450334 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.450353 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.451646 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.451649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.452861 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.452873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.453394 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.453399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.453639 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.454256 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.454272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.456086 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.458087 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.458092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.458106 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.458456 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.458750 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.458753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.458755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.458898 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.459345 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.459354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.459699 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.459911 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.460220 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260620 21:55:50.462710 25408 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39359
I20260620 21:55:50.462848 25408 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:50.463397 25408 heartbeater.cc:511] Master 127.17.67.254:39359 requested a full tablet report, sending...
I20260620 21:55:50.465014 25204 ts_manager.cc:194] Registered new tserver with Master: f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035)
I20260620 21:55:50.466706 25204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:42943
WARNING: no policy specified for oryx/127.17.67.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.194@KRBTEST.COM" created.
I20260620 21:55:50.472587 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.464998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:32913 (local address 127.17.67.254:39359)
0620 21:55:50.465107 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.465111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.465342 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:55:50.465464 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.465466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.465507 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.465619 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.466034 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.466538 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.467661 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.467963 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.468078 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.469418 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.469420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.469422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.469442 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.470956 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.471389 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.471391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.471392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.471461 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.471777 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.471780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.471783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.471958 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.472039 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.472306 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.472399 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:55:50.472783 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.464945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.193:32913)
0620 21:55:50.465257 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.465269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.465380 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.465635 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.465638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.465831 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.465922 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.465927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.466660 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.466663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.467229 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.467237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.468098 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.468103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.468271 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.468811 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.468830 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.469282 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.471073 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.471076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.471078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.471293 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.471538 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.471541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.471543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.471623 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.472038 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.472040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.472117 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.472243 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.472411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":25,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
I20260620 21:55:50.488663 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39359
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:50.599086 25417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:50.599347 25417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:50.599393 25417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:50.603498 25417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:50.603557 25417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:50.603602 25417 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:50.603667 25417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:50.609076 25417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:50.610229 25417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:50.611299 25417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:50.621312 25422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.621327 25423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.621428 25417 server_base.cc:1034] running on GCE node
W20260620 21:55:50.621327 25425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.622018 25417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:50.622694 25417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:50.623893 25417 hybrid_clock.cc:648] HybridClock initialized: now 1781992550623872 us; error 32 us; skew 500 ppm
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:50.626796 25417 init.cc:377] Logged in from keytab as oryx/127.17.67.194@KRBTEST.COM (short username oryx)
I20260620 21:55:50.627774 25417 webserver.cc:460] Webserver started at http://127.17.67.194:36937/ using document root <none> and password file <none>
I20260620 21:55:50.628340 25417 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:50.628434 25417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:50.628672 25417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:50.630801 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "157e880aa537451b8d60d9d089f07f82"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "5b074529d53c2df108145d803d8d6358"
server_key_iv: "2600f05dd5cb420937f4c1399aace6d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.631372 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "157e880aa537451b8d60d9d089f07f82"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "5b074529d53c2df108145d803d8d6358"
server_key_iv: "2600f05dd5cb420937f4c1399aace6d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.635300 25417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:55:50.638094 25432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.639361 25417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:55:50.639520 25417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "157e880aa537451b8d60d9d089f07f82"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "5b074529d53c2df108145d803d8d6358"
server_key_iv: "2600f05dd5cb420937f4c1399aace6d9"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.639679 25417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:50.661325 25417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:50.664698 25417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:50.664916 25417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:50.666471 25417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:50.666546 25417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.666659 25417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:50.666713 25417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:50.677896 25417 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:33565
I20260620 21:55:50.677970 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:33565 every 8 connection(s)
I20260620 21:55:50.679061 25417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:50.680944 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.667400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54051 (local address 127.17.67.254:39359)
0620 21:55:50.667548 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.667552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.668606 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 21:55:50.669784 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.669788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.669850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.669961 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.671195 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.671723 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.672983 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.673275 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.673390 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.677035 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.677039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.677042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.677073 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.678856 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.679467 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.679471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.679474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.679573 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.679953 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.679956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.679958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.680158 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.680238 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.680731 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.680830 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:50.681859 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.667916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.194:54051)
0620 21:55:50.668445 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.668487 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.669500 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.670138 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.670144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.670554 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.671042 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.671055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.671849 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.671855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.672862 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.672871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.673438 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.673442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.673658 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.674358 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.674383 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.676858 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.678976 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.678981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.678992 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.679334 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.679675 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.679678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.679680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.679844 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.680261 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.680267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.680494 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.680653 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.680924 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":173,"threads_started":1}
I20260620 21:55:50.682984 25548 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39359
I20260620 21:55:50.683138 25548 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:50.683617 25548 heartbeater.cc:511] Master 127.17.67.254:39359 requested a full tablet report, sending...
I20260620 21:55:50.684883 25205 ts_manager.cc:194] Registered new tserver with Master: 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:50.685649 25205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:54051
I20260620 21:55:50.685801 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25417
I20260620 21:55:50.685930 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:50.686239 17679 external_mini_cluster.cc:1428] Setting key 712d6f03ff1607db223e77aa17a74972
I20260620 21:55:50.694924 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.686677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54307 (local address 127.17.67.254:39359)
0620 21:55:50.686919 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.686924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.686945 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:55:50.686975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.686977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.687011 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.687109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.687929 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.688707 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.689685 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.689895 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.689975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.691429 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.691432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.691435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.691458 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.693147 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.693681 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.693683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.693684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.693756 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.694097 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.694100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.694101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.694307 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.694391 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.694710 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.694811 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260620 21:55:50.695055 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.685683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.194:54307)
0620 21:55:50.686084 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.686100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.686194 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.687143 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.687146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.687382 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.687669 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.687677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.688845 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.688847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.689528 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.689541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.690010 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.690015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.690129 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.690798 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.690817 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.691287 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.693261 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.693267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.693271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.693557 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.693851 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.693856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.693861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.693970 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.694416 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.694421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.694530 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.694635 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.694800 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for oryx/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
I20260620 21:55:50.714772 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39359
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:50.825886 25557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:50.826148 25557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:50.826253 25557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:50.830508 25557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:50.830590 25557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:50.830631 25557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:55:50.830708 25557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:50.835980 25557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.67.254:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:50.837059 25557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:50.838099 25557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:50.848001 25563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.848001 25562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:50.848131 25565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:50.848572 25557 server_base.cc:1034] running on GCE node
I20260620 21:55:50.848950 25557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:50.849517 25557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:50.850739 25557 hybrid_clock.cc:648] HybridClock initialized: now 1781992550850724 us; error 26 us; skew 500 ppm
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:50.853462 25557 init.cc:377] Logged in from keytab as oryx/127.17.67.195@KRBTEST.COM (short username oryx)
I20260620 21:55:50.854429 25557 webserver.cc:460] Webserver started at http://127.17.67.195:38725/ using document root <none> and password file <none>
I20260620 21:55:50.854949 25557 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:50.855008 25557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:50.855173 25557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:50.857167 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "f53c1cc5fc11440eb80482a568c43840"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "063bb698b7e5fe597bcd14c977c7d6cf"
server_key_iv: "6f9328c90046ca7ce9830c39fbefa8e7"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.857744 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "f53c1cc5fc11440eb80482a568c43840"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "063bb698b7e5fe597bcd14c977c7d6cf"
server_key_iv: "6f9328c90046ca7ce9830c39fbefa8e7"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.861322 25557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:50.864012 25572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.865270 25557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:55:50.865406 25557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "f53c1cc5fc11440eb80482a568c43840"
format_stamp: "Formatted at 2026-06-20 21:55:50 on dist-test-slave-cx8d"
server_key: "063bb698b7e5fe597bcd14c977c7d6cf"
server_key_iv: "6f9328c90046ca7ce9830c39fbefa8e7"
server_key_version: "encryptionkey@0"
I20260620 21:55:50.865516 25557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:50.876991 25557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:50.880132 25557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:50.880334 25557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:50.881817 25557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:50.881891 25557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:50.881961 25557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:50.882011 25557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:50.892999 25557 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:34517
I20260620 21:55:50.893059 25687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:34517 every 8 connection(s)
I20260620 21:55:50.894188 25557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:50.895895 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.882603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:51277 (local address 127.17.67.254:39359)
0620 21:55:50.882750 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.882755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.883833 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 21:55:50.885118 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.885123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.885196 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.885290 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.886919 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.887530 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.888750 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.888978 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.889090 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.892004 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.892007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.892009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.892033 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.893882 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.894361 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.894364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.894366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.894453 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.894895 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.894898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.894900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.895103 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.895185 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.895674 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.895778 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:55:50.896672 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.883092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.195:51277)
0620 21:55:50.883657 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.883719 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.884857 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.885482 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.885494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.885996 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.886741 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.886764 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.887662 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.887667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.888630 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.888639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.889137 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.889140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.889308 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.889976 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.889995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.891833 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.893985 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.893989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.894000 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.894250 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.894573 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.894577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.894580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.894778 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.895218 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.895224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.895434 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.895589 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.895834 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260620 21:55:50.897933 25688 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39359
I20260620 21:55:50.898074 25688 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:50.898631 25688 heartbeater.cc:511] Master 127.17.67.254:39359 requested a full tablet report, sending...
I20260620 21:55:50.899753 25204 ts_manager.cc:194] Registered new tserver with Master: f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517)
I20260620 21:55:50.900344 25204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:51277
I20260620 21:55:50.901248 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25557
I20260620 21:55:50.901398 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:50.901755 17679 external_mini_cluster.cc:1428] Setting key 2c119cb29dcfd47351e73ee35dedfce5
I20260620 21:55:50.903887 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:55:50.908622 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.900330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.17.67.195:34783)
0620 21:55:50.900675 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.900692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.900789 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.901034 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.901037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.901260 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:50.901388 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.901395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.902341 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.902345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.903173 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.903181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.903612 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.903615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.903712 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.904271 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:50.904285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:50.904734 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:50.907122 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.907125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.907127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.907326 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.907634 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:50.907636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:50.907637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:50.907738 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:50.908220 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:50.908222 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:50.908293 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.908371 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.908483 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260620 21:55:50.908841 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.900380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:34783 (local address 127.17.67.254:39359)
0620 21:55:50.900497 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.900501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.900765 (+ 264us) server_negotiation.cc:407] Connection header received
0620 21:55:50.900871 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.900873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.900911 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.900976 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.901539 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.902216 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.903295 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.903504 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.903608 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.904924 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.904927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.904930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.904954 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.906988 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.907428 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.907435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.907440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.907536 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.907830 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.907837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.907842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.908120 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.908234 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.908464 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.908610 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:55:50.930964 25254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:50.924195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.17.67.254:39359)
0620 21:55:50.924334 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.924338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.924352 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:50.924395 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.924398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.924437 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.924514 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.925243 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.925906 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.926904 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.927217 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.927301 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.930754 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57374: BlockingRecv error: recv got EOF from 127.0.0.1:57374 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 21:55:50 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:50.940660 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.932900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.17.67.254:39359)
0620 21:55:50.933020 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.933024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.933167 (+ 143us) server_negotiation.cc:407] Connection header received
0620 21:55:50.933307 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.933310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.933352 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.933427 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:50.934065 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.934555 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.935331 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.935528 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.935601 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.937900 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:50.937903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:50.937905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:50.937927 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:50.939343 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.939711 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.939713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.939715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.939784 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:50.940007 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:50.940009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:50.940010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:50.940164 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:50.940259 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.940427 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.940510 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:50.943709 25204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:50.947037 25204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:50.961242 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.955625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:34517 (local address 127.0.0.1:48382)
0620 21:55:50.956178 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.956230 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.956349 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.957156 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.957163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.957188 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:50.957333 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.957340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.958657 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.958660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.959933 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.959942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.960715 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.960718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.960825 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.960863 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.960979 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.961096 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260620 21:55:50.961254 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.955799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.17.67.195:34517)
0620 21:55:50.956643 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.956655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.956679 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:55:50.956754 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.956759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.956966 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.957488 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:50.957651 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.958538 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.960049 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.960593 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.960718 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.960885 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.961009 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.961096 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":108,"threads_started":1}
I20260620 21:55:50.962729 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.955819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33565 (local address 127.0.0.1:36074)
0620 21:55:50.956549 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.956566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.956648 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.957679 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.957682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.957716 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:50.957884 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.957892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.959264 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.959268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.960980 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.960991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.962196 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.962199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.962295 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.962317 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.962528 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.962599 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":234,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260620 21:55:50.962957 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.956338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46035 (local address 127.0.0.1:58780)
0620 21:55:50.956811 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:50.956824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:50.956893 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:50.957679 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:50.957682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:50.957708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:50.957843 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.957850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.959281 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.959284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.960980 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:50.960991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.962391 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.962400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:50.962507 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.962529 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:55:50.962595 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:55:50.962665 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":377,"spinlock_wait_cycles":1408,"thread_start_us":94,"threads_started":1}
I20260620 21:55:50.963142 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.956431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.17.67.194:33565)
0620 21:55:50.957199 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.957211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.957231 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:50.957289 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.957292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.957503 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.957828 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:50.957971 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.959124 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.961122 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.962095 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.962192 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.962343 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.962903 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.963000 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":132,"threads_started":1}
I20260620 21:55:50.963496 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:50.956765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.17.67.193:46035)
0620 21:55:50.957113 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:50.957126 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:50.957146 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:50.957210 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:50.957214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:50.957447 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:50.957613 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:50.957973 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.959158 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.961122 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:50.962062 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:50.962176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:50.962323 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:50.963268 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 21:55:50.963373 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
I20260620 21:55:50.964395 25622 tablet_service.cc:1467] Processing CreateTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 (DEFAULT_TABLE table=test-table [id=4e0859ed5b724d36ac7b84491b679d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:50.965540 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a966d58a2ed647d89221cf9e5fc7a4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:50.965824 25482 tablet_service.cc:1467] Processing CreateTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 (DEFAULT_TABLE table=test-table [id=4e0859ed5b724d36ac7b84491b679d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:50.966854 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a966d58a2ed647d89221cf9e5fc7a4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:50.967924 25342 tablet_service.cc:1467] Processing CreateTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 (DEFAULT_TABLE table=test-table [id=4e0859ed5b724d36ac7b84491b679d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:50.968858 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a966d58a2ed647d89221cf9e5fc7a4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:50.974323 25716 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Bootstrap starting.
I20260620 21:55:50.975307 25718 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Bootstrap starting.
I20260620 21:55:50.975414 25717 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Bootstrap starting.
I20260620 21:55:50.976672 25716 tablet_bootstrap.cc:654] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:50.977110 25718 tablet_bootstrap.cc:654] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:50.977331 25717 tablet_bootstrap.cc:654] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:50.977537 25716 log.cc:824] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:50.977934 25718 log.cc:824] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:50.978549 25717 log.cc:824] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:50.979568 25716 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: No bootstrap required, opened a new log
I20260620 21:55:50.979760 25716 ts_tablet_manager.cc:1397] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:55:50.979912 25718 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: No bootstrap required, opened a new log
I20260620 21:55:50.980087 25718 ts_tablet_manager.cc:1397] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:55:50.980417 25717 tablet_bootstrap.cc:492] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: No bootstrap required, opened a new log
I20260620 21:55:50.980608 25717 ts_tablet_manager.cc:1397] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 21:55:50.983484 25717 raft_consensus.cc:348] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.983733 25717 raft_consensus.cc:374] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:50.983798 25717 raft_consensus.cc:729] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f2240aef90489bb774464c943b8237, State: Initialized, Role: FOLLOWER
I20260620 21:55:50.983889 25716 raft_consensus.cc:348] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.983878 25718 raft_consensus.cc:348] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.984131 25718 raft_consensus.cc:374] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:50.984140 25716 raft_consensus.cc:374] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:50.984190 25716 raft_consensus.cc:729] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Initialized, Role: FOLLOWER
I20260620 21:55:50.984190 25718 raft_consensus.cc:729] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Initialized, Role: FOLLOWER
I20260620 21:55:50.984287 25717 consensus_queue.cc:260] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.984583 25716 consensus_queue.cc:260] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.984583 25718 consensus_queue.cc:260] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:50.984946 25408 heartbeater.cc:503] Master 127.17.67.254:39359 was elected leader, sending a full tablet report...
I20260620 21:55:50.985271 25717 ts_tablet_manager.cc:1428] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:55:50.985507 25716 ts_tablet_manager.cc:1428] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 21:55:50.985673 25718 ts_tablet_manager.cc:1428] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:55:50.985828 25688 heartbeater.cc:503] Master 127.17.67.254:39359 was elected leader, sending a full tablet report...
I20260620 21:55:50.986922 25548 heartbeater.cc:503] Master 127.17.67.254:39359 was elected leader, sending a full tablet report...
I20260620 21:55:51.042970 25724 raft_consensus.cc:482] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:51.043226 25724 raft_consensus.cc:504] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:51.044571 25724 leader_election.cc:288] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517), f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035)
I20260620 21:55:51.048528 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.044890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:34517 (local address 127.17.67.194:59719)
0620 21:55:51.045038 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.045055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.045194 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.045492 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.045498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.045520 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.045675 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.045681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.046547 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.046554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.047527 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.047537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.048163 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.048168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.048263 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.048283 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.048348 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.048410 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 21:55:51.048580 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.044975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:59719 (local address 127.17.67.195:34517)
0620 21:55:51.045134 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.045139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.045158 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:55:51.045303 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.045312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.045378 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.045457 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.045805 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.046414 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.047663 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.048047 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.048177 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.048231 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.048381 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.048447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:55:51.049248 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.045161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46035 (local address 127.17.67.194:55845)
0620 21:55:51.045556 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.045569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.045677 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.045959 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.045961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.045974 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.046406 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.046412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.047155 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.047158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.048046 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.048054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.048910 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.048913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.049017 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.049040 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.049099 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.049146 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":143,"threads_started":1}
I20260620 21:55:51.049222 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a966d58a2ed647d89221cf9e5fc7a4a6" candidate_uuid: "157e880aa537451b8d60d9d089f07f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c1cc5fc11440eb80482a568c43840" is_pre_election: true
I20260620 21:55:51.049319 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.045294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:55845 (local address 127.17.67.193:46035)
0620 21:55:51.045450 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.045455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.045653 (+ 198us) server_negotiation.cc:407] Connection header received
0620 21:55:51.045813 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.045816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.045864 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.045942 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.046529 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.047035 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.048187 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.048779 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.048898 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.048953 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.049138 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.049204 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:55:51.049603 25642 raft_consensus.cc:2454] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e880aa537451b8d60d9d089f07f82 in term 0.
I20260620 21:55:51.049917 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a966d58a2ed647d89221cf9e5fc7a4a6" candidate_uuid: "157e880aa537451b8d60d9d089f07f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237" is_pre_election: true
I20260620 21:55:51.050117 25434 leader_election.cc:302] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e880aa537451b8d60d9d089f07f82, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.050230 25362 raft_consensus.cc:2454] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e880aa537451b8d60d9d089f07f82 in term 0.
I20260620 21:55:51.050397 25724 raft_consensus.cc:2781] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:51.050503 25724 raft_consensus.cc:482] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:51.050572 25724 raft_consensus.cc:3037] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.051749 25724 raft_consensus.cc:504] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:51.052078 25724 leader_election.cc:288] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [CANDIDATE]: Term 1 election: Requested vote from peers f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517), f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035)
I20260620 21:55:51.052423 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a966d58a2ed647d89221cf9e5fc7a4a6" candidate_uuid: "157e880aa537451b8d60d9d089f07f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c1cc5fc11440eb80482a568c43840"
I20260620 21:55:51.052513 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a966d58a2ed647d89221cf9e5fc7a4a6" candidate_uuid: "157e880aa537451b8d60d9d089f07f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237"
I20260620 21:55:51.052567 25642 raft_consensus.cc:3037] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.052659 25362 raft_consensus.cc:3037] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.053870 25642 raft_consensus.cc:2454] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e880aa537451b8d60d9d089f07f82 in term 1.
I20260620 21:55:51.054008 25362 raft_consensus.cc:2454] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e880aa537451b8d60d9d089f07f82 in term 1.
I20260620 21:55:51.054225 25434 leader_election.cc:302] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e880aa537451b8d60d9d089f07f82, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.054418 25724 raft_consensus.cc:2781] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:51.054673 25724 raft_consensus.cc:686] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 1 LEADER]: Becoming Leader. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Running, Role: LEADER
I20260620 21:55:51.054983 25724 consensus_queue.cc:237] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } }
I20260620 21:55:51.058012 25204 catalog_manager.cc:5495] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 157e880aa537451b8d60d9d089f07f82 (127.17.67.194). New cstate: current_term: 1 leader_uuid: "157e880aa537451b8d60d9d089f07f82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:51.085395 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.081653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.17.67.194:33565)
0620 21:55:51.081796 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.081803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.081818 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:51.082021 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.082023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.082076 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.082202 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:51.082439 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.083008 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.083875 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.084167 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.084278 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.084554 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.084983 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.085120 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.085183 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.085238 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:55:51.091279 25362 raft_consensus.cc:1261] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Refusing update from remote peer 157e880aa537451b8d60d9d089f07f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.091477 25642 raft_consensus.cc:1261] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Refusing update from remote peer 157e880aa537451b8d60d9d089f07f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.092046 25726 consensus_queue.cc:1035] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.092262 25724 consensus_queue.cc:1035] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.100668 25731 mvcc.cc:205] Tried to move back new op lower bound from 7299041489261703168 to 7299041489123704832. Current Snapshot: MvccSnapshot[applied={T|T < 7299041489261703168}]
I20260620 21:55:51.107611 25205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.107784 25205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.110136 25205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Sending DeleteTablet for 3 replicas of tablet a966d58a2ed647d89221cf9e5fc7a4a6
I20260620 21:55:51.110886 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:36074
I20260620 21:55:51.110970 25342 tablet_service.cc:1514] Processing DeleteTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 21:55:51.111382 25739 tablet_replica.cc:330] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: stopping tablet replica
I20260620 21:55:51.111378 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet a966d58a2ed647d89221cf9e5fc7a4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:48382
I20260620 21:55:51.111706 25739 raft_consensus.cc:2229] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:51.111891 25741 tablet_replica.cc:330] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: stopping tablet replica
I20260620 21:55:51.112092 25739 raft_consensus.cc:2258] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.112275 25741 raft_consensus.cc:2229] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:51.112574 25740 tablet_replica.cc:330] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: stopping tablet replica
I20260620 21:55:51.112601 25741 raft_consensus.cc:2258] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.112632 25205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:55:51.112851 25740 raft_consensus.cc:2229] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 21:55:51.113049 25205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:51.113116 25740 raft_consensus.cc:2258] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.114516 25741 ts_tablet_manager.cc:1905] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.114516 25740 ts_tablet_manager.cc:1905] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.117231 25740 ts_tablet_manager.cc:1918] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.117233 25741 ts_tablet_manager.cc:1918] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.117345 25740 log.cc:1196] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/a966d58a2ed647d89221cf9e5fc7a4a6
I20260620 21:55:51.117449 25741 log.cc:1196] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/a966d58a2ed647d89221cf9e5fc7a4a6
I20260620 21:55:51.117684 25740 ts_tablet_manager.cc:1939] T a966d58a2ed647d89221cf9e5fc7a4a6 P f4f2240aef90489bb774464c943b8237: Deleting consensus metadata
I20260620 21:55:51.117921 25741 ts_tablet_manager.cc:1939] T a966d58a2ed647d89221cf9e5fc7a4a6 P f53c1cc5fc11440eb80482a568c43840: Deleting consensus metadata
I20260620 21:55:51.118901 25342 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d471773444bbca38cc5941b0464bb (DEFAULT_TABLE table=test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.118992 25622 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d471773444bbca38cc5941b0464bb (DEFAULT_TABLE table=test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.119038 25190 catalog_manager.cc:4841] TS f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517): tablet a966d58a2ed647d89221cf9e5fc7a4a6 (table test-table [id=4e0859ed5b724d36ac7b84491b679d61]) successfully deleted
I20260620 21:55:51.119130 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d471773444bbca38cc5941b0464bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.119234 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d471773444bbca38cc5941b0464bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.120819 25716 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Bootstrap starting.
I20260620 21:55:51.121083 25717 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Bootstrap starting.
I20260620 21:55:51.121483 25482 tablet_service.cc:1467] Processing CreateTablet for tablet 3d9d471773444bbca38cc5941b0464bb (DEFAULT_TABLE table=test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.121753 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d9d471773444bbca38cc5941b0464bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.121768 25716 tablet_bootstrap.cc:654] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.122022 25717 tablet_bootstrap.cc:654] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.122298 25739 ts_tablet_manager.cc:1905] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.123149 25717 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: No bootstrap required, opened a new log
I20260620 21:55:51.123234 25717 ts_tablet_manager.cc:1397] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:55:51.123673 25717 raft_consensus.cc:348] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.123773 25717 raft_consensus.cc:374] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.123811 25717 raft_consensus.cc:729] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f2240aef90489bb774464c943b8237, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.123930 25717 consensus_queue.cc:260] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.124176 25717 ts_tablet_manager.cc:1428] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:51.124723 25739 ts_tablet_manager.cc:1918] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.124815 25739 log.cc:1196] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/a966d58a2ed647d89221cf9e5fc7a4a6
I20260620 21:55:51.124835 25718 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Bootstrap starting.
I20260620 21:55:51.125105 25739 ts_tablet_manager.cc:1939] T a966d58a2ed647d89221cf9e5fc7a4a6 P 157e880aa537451b8d60d9d089f07f82: Deleting consensus metadata
I20260620 21:55:51.125733 25718 tablet_bootstrap.cc:654] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.126371 25190 catalog_manager.cc:4841] TS f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035): tablet a966d58a2ed647d89221cf9e5fc7a4a6 (table test-table [id=4e0859ed5b724d36ac7b84491b679d61]) successfully deleted
I20260620 21:55:51.127053 25191 catalog_manager.cc:4841] TS 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565): tablet a966d58a2ed647d89221cf9e5fc7a4a6 (table test-table [id=4e0859ed5b724d36ac7b84491b679d61]) successfully deleted
I20260620 21:55:51.127534 25716 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: No bootstrap required, opened a new log
I20260620 21:55:51.127626 25716 ts_tablet_manager.cc:1397] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 21:55:51.128036 25716 raft_consensus.cc:348] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.128082 25718 tablet_bootstrap.cc:492] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: No bootstrap required, opened a new log
I20260620 21:55:51.128118 25716 raft_consensus.cc:374] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.128139 25716 raft_consensus.cc:729] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.128150 25718 ts_tablet_manager.cc:1397] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 21:55:51.128274 25716 consensus_queue.cc:260] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.128531 25716 ts_tablet_manager.cc:1428] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:51.128521 25718 raft_consensus.cc:348] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.128613 25718 raft_consensus.cc:374] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.128643 25718 raft_consensus.cc:729] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.128758 25718 consensus_queue.cc:260] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.129004 25718 ts_tablet_manager.cc:1428] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 21:55:51.145213 25689 tablet.cc:2369] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:55:51.180234 25549 tablet.cc:2369] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:55:51.209496 25409 tablet.cc:2369] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:51.326318 25723 raft_consensus.cc:482] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:51.326467 25723 raft_consensus.cc:504] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.327576 25723 leader_election.cc:288] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:51.332502 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.327967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:37475 (local address 127.17.67.193:46035)
0620 21:55:51.328099 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.328103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.328412 (+ 309us) server_negotiation.cc:407] Connection header received
0620 21:55:51.328500 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.328502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.328539 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.328597 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.328876 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.329605 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331132 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.331749 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.331951 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.332298 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.332374 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:55:51.332502 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.327951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:40759 (local address 127.17.67.194:33565)
0620 21:55:51.328093 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.328097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.328134 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:51.328199 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.328202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.328258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.328335 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.328680 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.329452 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331132 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.331749 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.331951 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.332298 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.332376 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:55:51.332536 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.327835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33565 (local address 127.17.67.195:40759)
0620 21:55:51.327983 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.328002 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.328108 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.328381 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.328384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.328409 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.328549 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.328554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.329585 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.329588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.330974 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.330985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.331905 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.332133 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.332176 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.332260 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.332362 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":138,"spinlock_wait_cycles":1792}
I20260620 21:55:51.332546 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.327835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46035 (local address 127.17.67.195:37475)
0620 21:55:51.328284 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.328294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.328417 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.328640 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.328643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.328662 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.328770 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.328775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.329721 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.329724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.330979 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.330985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.331919 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.331922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.332128 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.332173 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.332260 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.332362 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":272,"spinlock_wait_cycles":2816,"thread_start_us":122,"threads_started":1}
I20260620 21:55:51.333112 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d471773444bbca38cc5941b0464bb" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82" is_pre_election: true
I20260620 21:55:51.333308 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d471773444bbca38cc5941b0464bb" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237" is_pre_election: true
I20260620 21:55:51.333388 25502 raft_consensus.cc:2454] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:51.333441 25362 raft_consensus.cc:2454] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:51.333842 25574 leader_election.cc:302] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 157e880aa537451b8d60d9d089f07f82, f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.334137 25723 raft_consensus.cc:2781] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:51.334197 25723 raft_consensus.cc:482] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:51.334235 25723 raft_consensus.cc:3037] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.335235 25723 raft_consensus.cc:504] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.335573 25723 leader_election.cc:288] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Requested vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:51.335909 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d471773444bbca38cc5941b0464bb" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237"
I20260620 21:55:51.335980 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d9d471773444bbca38cc5941b0464bb" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82"
I20260620 21:55:51.336055 25362 raft_consensus.cc:3037] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.336113 25502 raft_consensus.cc:3037] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.336956 25362 raft_consensus.cc:2454] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:51.337107 25502 raft_consensus.cc:2454] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:51.337257 25574 leader_election.cc:302] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.337422 25723 raft_consensus.cc:2781] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:51.337687 25723 raft_consensus.cc:686] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 1 LEADER]: Becoming Leader. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Running, Role: LEADER
I20260620 21:55:51.337941 25723 consensus_queue.cc:237] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.340107 25205 catalog_manager.cc:5495] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53c1cc5fc11440eb80482a568c43840 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } health_report { overall_health: HEALTHY } } }
Jun 20 21:55:51 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992550, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.254@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:51.387709 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.379345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.17.67.254:39359)
0620 21:55:51.379499 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.379504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.379683 (+ 179us) server_negotiation.cc:407] Connection header received
0620 21:55:51.379863 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.379866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.379912 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.379997 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:51.380691 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.381215 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.382060 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.382274 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.382382 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.384680 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:51.384684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:51.384688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:51.384716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:51.386122 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:51.386666 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:51.386669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:51.386670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:51.386742 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:51.386969 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:51.386971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:51.386972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:51.387131 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:51.387215 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.387461 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.387576 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":55}
I20260620 21:55:51.387717 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.379297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39359 (local address 127.0.0.1:57402)
0620 21:55:51.379594 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.379606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.379764 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.380013 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.380015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.380475 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:51.380566 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.380574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.381331 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.381335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.381949 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.381957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.382380 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.382384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.382453 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.383005 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:51.383017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:51.384546 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:51.386268 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:51.386277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:51.386300 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:51.386552 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:51.386818 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:51.386820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:51.386821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:51.386898 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:51.387214 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:51.387219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:51.387319 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.387408 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.387522 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
I20260620 21:55:51.391472 25205 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:57402:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:55:51.392186 25205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:51.397089 25342 tablet_service.cc:1467] Processing CreateTablet for tablet 8360ada133544938942ec714c22d317f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2ea355a04ed4ea88784c32da55ac147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:51.397126 25482 tablet_service.cc:1467] Processing CreateTablet for tablet 8360ada133544938942ec714c22d317f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2ea355a04ed4ea88784c32da55ac147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:51.397089 25622 tablet_service.cc:1467] Processing CreateTablet for tablet 8360ada133544938942ec714c22d317f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2ea355a04ed4ea88784c32da55ac147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:55:51.397477 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8360ada133544938942ec714c22d317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.397477 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8360ada133544938942ec714c22d317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.397497 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8360ada133544938942ec714c22d317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.399989 25718 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: Bootstrap starting.
I20260620 21:55:51.400282 25716 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: Bootstrap starting.
I20260620 21:55:51.400903 25718 tablet_bootstrap.cc:654] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.401201 25716 tablet_bootstrap.cc:654] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.401739 25717 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: Bootstrap starting.
I20260620 21:55:51.402127 25718 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: No bootstrap required, opened a new log
I20260620 21:55:51.402220 25718 ts_tablet_manager.cc:1397] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 21:55:51.402611 25718 raft_consensus.cc:348] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.402685 25717 tablet_bootstrap.cc:654] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.402733 25718 raft_consensus.cc:374] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.402774 25718 raft_consensus.cc:729] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.402801 25716 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: No bootstrap required, opened a new log
I20260620 21:55:51.402879 25716 ts_tablet_manager.cc:1397] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:51.402910 25718 consensus_queue.cc:260] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.403174 25718 ts_tablet_manager.cc:1428] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:51.403393 25716 raft_consensus.cc:348] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.403515 25716 raft_consensus.cc:374] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.403548 25716 raft_consensus.cc:729] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.403661 25716 consensus_queue.cc:260] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.403815 25736 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.403934 25716 ts_tablet_manager.cc:1428] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:51.403951 25736 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.403895 25723 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.404038 25723 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.404814 25717 tablet_bootstrap.cc:492] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: No bootstrap required, opened a new log
I20260620 21:55:51.404899 25717 ts_tablet_manager.cc:1397] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 21:55:51.405402 25717 raft_consensus.cc:348] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.405519 25717 raft_consensus.cc:374] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.405584 25717 raft_consensus.cc:729] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f2240aef90489bb774464c943b8237, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.405720 25717 consensus_queue.cc:260] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.405997 25717 ts_tablet_manager.cc:1428] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:51.406014 25722 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.406213 25722 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.633638 25723 raft_consensus.cc:482] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:51.633805 25723 raft_consensus.cc:504] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.634289 25723 leader_election.cc:288] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:51.634624 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8360ada133544938942ec714c22d317f" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237" is_pre_election: true
I20260620 21:55:51.634667 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8360ada133544938942ec714c22d317f" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82" is_pre_election: true
I20260620 21:55:51.634793 25362 raft_consensus.cc:2454] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:51.634810 25502 raft_consensus.cc:2454] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:51.635080 25574 leader_election.cc:302] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.635274 25723 raft_consensus.cc:2781] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:51.635335 25723 raft_consensus.cc:482] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:51.635372 25723 raft_consensus.cc:3037] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.636303 25723 raft_consensus.cc:504] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.636619 25723 leader_election.cc:288] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Requested vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:51.636950 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8360ada133544938942ec714c22d317f" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237"
I20260620 21:55:51.636989 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8360ada133544938942ec714c22d317f" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82"
I20260620 21:55:51.637059 25362 raft_consensus.cc:3037] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.637126 25502 raft_consensus.cc:3037] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.638036 25362 raft_consensus.cc:2454] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:51.638144 25502 raft_consensus.cc:2454] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:51.638289 25574 leader_election.cc:302] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:51.638455 25723 raft_consensus.cc:2781] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:51.638542 25723 raft_consensus.cc:686] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [term 1 LEADER]: Becoming Leader. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Running, Role: LEADER
I20260620 21:55:51.638703 25723 consensus_queue.cc:237] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.639283 25746 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: TxnStatusTablet state changed. Reason: New leader f53c1cc5fc11440eb80482a568c43840. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.639397 25746 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:51.639600 25758 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:55:51.639760 25758 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:55:51.640380 25203 catalog_manager.cc:5495] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53c1cc5fc11440eb80482a568c43840 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } health_report { overall_health: HEALTHY } } }
I20260620 21:55:51.700574 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.696675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.17.67.195:34517)
0620 21:55:51.696911 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.696917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.696933 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:55:51.697019 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.697021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.697072 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.697200 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:51.697475 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.698124 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.699104 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.699348 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.699460 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.699763 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.700170 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.700294 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.700362 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.700414 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":63,"threads_started":1}
I20260620 21:55:51.700558 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.696555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:34517 (local address 127.0.0.1:48394)
0620 21:55:51.696728 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.696747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.696861 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.697206 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.697208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.697216 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:51.697357 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.697363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.698236 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.698242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.698991 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.699000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.699460 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.699463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.699553 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.699607 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.700268 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.700282 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.700353 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.700437 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:55:51.703908 25362 raft_consensus.cc:1261] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Refusing update from remote peer f53c1cc5fc11440eb80482a568c43840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.703962 25502 raft_consensus.cc:1261] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Refusing update from remote peer f53c1cc5fc11440eb80482a568c43840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.704439 25723 consensus_queue.cc:1035] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.704629 25746 consensus_queue.cc:1035] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.706130 25722 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: TxnStatusTablet state changed. Reason: New leader f53c1cc5fc11440eb80482a568c43840. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.706248 25722 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.706310 25736 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: TxnStatusTablet state changed. Reason: New leader f53c1cc5fc11440eb80482a568c43840. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.706421 25736 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.709023 25746 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.709152 25746 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:51.709280 25746 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.709355 25746 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f53c1cc5fc11440eb80482a568c43840: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:55:51.709335 25764 mvcc.cc:205] Tried to move back new op lower bound from 7299041491773685760 to 7299041491513270272. Current Snapshot: MvccSnapshot[applied={T|T < 7299041491773685760}]
I20260620 21:55:51.711413 25722 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.711537 25722 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P f4f2240aef90489bb774464c943b8237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.709512 25736 tablet_replica.cc:439] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } } }
I20260620 21:55:51.712457 25736 tablet_replica.cc:442] T 8360ada133544938942ec714c22d317f P 157e880aa537451b8d60d9d089f07f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:55:51.714807 25763 mvcc.cc:205] Tried to move back new op lower bound from 7299041491773685760 to 7299041491513270272. Current Snapshot: MvccSnapshot[applied={T|T < 7299041491773685760}]
I20260620 21:55:51.719472 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.714872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.17.67.195:34517)
0620 21:55:51.715006 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.715010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.715243 (+ 233us) server_negotiation.cc:407] Connection header received
0620 21:55:51.715409 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.715412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.715465 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.715514 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:51.715982 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.717099 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.718436 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.718641 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.718729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.719010 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.719072 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.719127 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.719258 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.719300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:55:51.725667 25723 consensus_queue.cc:1035] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.729673 25723 consensus_queue.cc:1035] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.732378 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.728104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:38003 (local address 127.17.67.195:34517)
0620 21:55:51.728218 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.728222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.728346 (+ 124us) server_negotiation.cc:407] Connection header received
0620 21:55:51.728483 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.728487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.728527 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.728571 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:51.728926 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.729783 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.731192 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.731509 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.731580 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.731941 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.732017 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.732056 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.732191 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.732240 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:55:51.732877 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.727426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:34517 (local address 127.17.67.195:38003)
0620 21:55:51.728248 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.728265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.728373 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.728711 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.728716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.728724 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:55:51.728818 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.728824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.729990 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.729993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.731061 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.731071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.731690 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.731694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.731773 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.731817 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.732104 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.732108 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.732667 (+ 559us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.732744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":98,"threads_started":1}
I20260620 21:55:51.777010 25205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.777149 25205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.779227 25205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Sending DeleteTablet for 3 replicas of tablet 3d9d471773444bbca38cc5941b0464bb
I20260620 21:55:51.779779 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9d471773444bbca38cc5941b0464bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:48382
I20260620 21:55:51.779894 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9d471773444bbca38cc5941b0464bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:36074
I20260620 21:55:51.780084 25342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d9d471773444bbca38cc5941b0464bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 21:55:51.780423 25781 tablet_replica.cc:330] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: stopping tablet replica
I20260620 21:55:51.780475 25779 tablet_replica.cc:330] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: stopping tablet replica
I20260620 21:55:51.780551 25781 raft_consensus.cc:2229] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:51.780575 25779 raft_consensus.cc:2229] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:51.780656 25781 raft_consensus.cc:2258] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.780857 25780 tablet_replica.cc:330] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: stopping tablet replica
I20260620 21:55:51.780857 25779 raft_consensus.cc:2258] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.780979 25780 raft_consensus.cc:2229] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:51.781081 25780 raft_consensus.cc:2258] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.781447 25779 ts_tablet_manager.cc:1905] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.784650 25781 ts_tablet_manager.cc:1905] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.786470 25779 ts_tablet_manager.cc:1918] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:51.786563 25779 log.cc:1196] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/3d9d471773444bbca38cc5941b0464bb
I20260620 21:55:51.786871 25779 ts_tablet_manager.cc:1939] T 3d9d471773444bbca38cc5941b0464bb P f53c1cc5fc11440eb80482a568c43840: Deleting consensus metadata
I20260620 21:55:51.787642 25781 ts_tablet_manager.cc:1918] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:51.787643 25190 catalog_manager.cc:4841] TS f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517): tablet 3d9d471773444bbca38cc5941b0464bb (table test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]) successfully deleted
I20260620 21:55:51.787726 25781 log.cc:1196] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/3d9d471773444bbca38cc5941b0464bb
I20260620 21:55:51.787982 25781 ts_tablet_manager.cc:1939] T 3d9d471773444bbca38cc5941b0464bb P f4f2240aef90489bb774464c943b8237: Deleting consensus metadata
I20260620 21:55:51.789235 25780 ts_tablet_manager.cc:1905] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.789494 25190 catalog_manager.cc:4841] TS f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035): tablet 3d9d471773444bbca38cc5941b0464bb (table test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]) successfully deleted
I20260620 21:55:51.791610 25780 ts_tablet_manager.cc:1918] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:51.791714 25780 log.cc:1196] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/3d9d471773444bbca38cc5941b0464bb
I20260620 21:55:51.791941 25780 ts_tablet_manager.cc:1939] T 3d9d471773444bbca38cc5941b0464bb P 157e880aa537451b8d60d9d089f07f82: Deleting consensus metadata
I20260620 21:55:51.792564 25191 catalog_manager.cc:4841] TS 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565): tablet 3d9d471773444bbca38cc5941b0464bb (table test-table [id=4db90fe8a1cb4bfc985ac09f54240b6e]) successfully deleted
Jun 20 21:55:51 dist-test-slave-cx8d krb5kdc[25160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:55:51 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:51 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:55:51.810977 25254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:51.803563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.17.67.254:39359)
0620 21:55:51.803723 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.803728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.803914 (+ 186us) server_negotiation.cc:407] Connection header received
0620 21:55:51.804114 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.804116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.804162 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.804235 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:51.805225 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.806150 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.807147 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.807533 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.807663 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.810831 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57418: BlockingRecv error: recv got EOF from 127.0.0.1:57418 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 21:55:51 dist-test-slave-cx8d krb5kdc[25160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM
I20260620 21:55:51.822428 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.812410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.17.67.254:39359)
0620 21:55:51.812528 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.812533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.812748 (+ 215us) server_negotiation.cc:407] Connection header received
0620 21:55:51.812906 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.812908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.812950 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.813023 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:51.814045 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.814536 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.815815 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.816051 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.816150 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.819034 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:51.819038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:51.819040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:51.819065 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:51.820746 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:51.821193 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:51.821196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:51.821197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:51.821270 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:51.821547 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:51.821552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:51.821553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:51.821764 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:51.821905 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.822142 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.822265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:55:51.825057 25205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:55:51.825491 25205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:51.830435 25342 tablet_service.cc:1467] Processing CreateTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 (DEFAULT_TABLE table=test-table [id=5872aebc15b246b1a1da929ebf283a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.830572 25482 tablet_service.cc:1467] Processing CreateTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 (DEFAULT_TABLE table=test-table [id=5872aebc15b246b1a1da929ebf283a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.830701 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f01a5016dc4742b6f7e25396e3d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.830876 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f01a5016dc4742b6f7e25396e3d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.831753 25622 tablet_service.cc:1467] Processing CreateTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 (DEFAULT_TABLE table=test-table [id=5872aebc15b246b1a1da929ebf283a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.832026 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f01a5016dc4742b6f7e25396e3d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.832556 25717 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Bootstrap starting.
I20260620 21:55:51.833542 25717 tablet_bootstrap.cc:654] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.834584 25716 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Bootstrap starting.
I20260620 21:55:51.834649 25717 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: No bootstrap required, opened a new log
I20260620 21:55:51.834728 25717 ts_tablet_manager.cc:1397] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:55:51.835114 25717 raft_consensus.cc:348] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.835230 25717 raft_consensus.cc:374] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.835264 25717 raft_consensus.cc:729] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f2240aef90489bb774464c943b8237, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.835388 25717 consensus_queue.cc:260] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.835466 25716 tablet_bootstrap.cc:654] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.835669 25717 ts_tablet_manager.cc:1428] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:51.837181 25718 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Bootstrap starting.
I20260620 21:55:51.838135 25718 tablet_bootstrap.cc:654] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.838809 25716 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: No bootstrap required, opened a new log
I20260620 21:55:51.838896 25716 ts_tablet_manager.cc:1397] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 21:55:51.839252 25716 raft_consensus.cc:348] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.839331 25716 raft_consensus.cc:374] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.839351 25716 raft_consensus.cc:729] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.839352 25718 tablet_bootstrap.cc:492] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: No bootstrap required, opened a new log
I20260620 21:55:51.839427 25718 ts_tablet_manager.cc:1397] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:55:51.839438 25716 consensus_queue.cc:260] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.839682 25716 ts_tablet_manager.cc:1428] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:51.839811 25718 raft_consensus.cc:348] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.839926 25718 raft_consensus.cc:374] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:51.839969 25718 raft_consensus.cc:729] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Initialized, Role: FOLLOWER
I20260620 21:55:51.840090 25718 consensus_queue.cc:260] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.840353 25718 ts_tablet_manager.cc:1428] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:55:51.942368 25722 raft_consensus.cc:482] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:51.942518 25722 raft_consensus.cc:504] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.943763 25722 leader_election.cc:288] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565), f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517)
I20260620 21:55:51.948812 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.944366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:58105 (local address 127.17.67.195:34517)
0620 21:55:51.944488 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.944493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.944509 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:55:51.944589 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.944593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.944642 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.944731 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.945040 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.945846 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.947603 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.948192 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.948337 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.948395 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.948585 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.948678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:51.949065 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.944114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:43035 (local address 127.17.67.194:33565)
0620 21:55:51.944634 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.944637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.944648 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:51.944730 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.944734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.944781 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.944863 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:55:51.945206 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.945776 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.947933 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.948433 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.948564 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.948612 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.948836 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.948920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":57,"threads_started":1}
I20260620 21:55:51.949061 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.943929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33565 (local address 127.17.67.193:43035)
0620 21:55:51.944287 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.944303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.944412 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.944967 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.944972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.944992 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.945091 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.945096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.945888 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.945891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.947823 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.947831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.948587 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.948591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.948694 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.948719 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.948806 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.948884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":128,"thread_start_us":86,"threads_started":1}
I20260620 21:55:51.949656 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.944030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:34517 (local address 127.17.67.193:58105)
0620 21:55:51.944367 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:51.944378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:51.944442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:51.944754 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:51.944757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:51.944774 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:55:51.944916 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.944923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.945950 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.945953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.947449 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:51.947465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.948325 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.948329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:51.948427 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.948455 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:55:51.949398 (+ 943us) client_negotiation.cc:240] Negotiation successful
0620 21:55:51.949479 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":104,"thread_start_us":67,"threads_started":1}
I20260620 21:55:51.949649 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f01a5016dc4742b6f7e25396e3d166" candidate_uuid: "f4f2240aef90489bb774464c943b8237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82" is_pre_election: true
I20260620 21:55:51.949827 25502 raft_consensus.cc:2454] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f2240aef90489bb774464c943b8237 in term 0.
I20260620 21:55:51.950219 25295 leader_election.cc:302] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e880aa537451b8d60d9d089f07f82, f4f2240aef90489bb774464c943b8237; no voters:
I20260620 21:55:51.950498 25722 raft_consensus.cc:2781] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:51.950453 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f01a5016dc4742b6f7e25396e3d166" candidate_uuid: "f4f2240aef90489bb774464c943b8237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c1cc5fc11440eb80482a568c43840" is_pre_election: true
I20260620 21:55:51.950582 25722 raft_consensus.cc:482] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:51.950630 25722 raft_consensus.cc:3037] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.950673 25642 raft_consensus.cc:2454] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f2240aef90489bb774464c943b8237 in term 0.
I20260620 21:55:51.951807 25722 raft_consensus.cc:504] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.952163 25722 leader_election.cc:288] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [CANDIDATE]: Term 1 election: Requested vote from peers 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565), f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517)
I20260620 21:55:51.952512 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f01a5016dc4742b6f7e25396e3d166" candidate_uuid: "f4f2240aef90489bb774464c943b8237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82"
I20260620 21:55:51.952636 25502 raft_consensus.cc:3037] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.953493 25502 raft_consensus.cc:2454] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f2240aef90489bb774464c943b8237 in term 1.
I20260620 21:55:51.953780 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f01a5016dc4742b6f7e25396e3d166" candidate_uuid: "f4f2240aef90489bb774464c943b8237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c1cc5fc11440eb80482a568c43840"
I20260620 21:55:51.953830 25295 leader_election.cc:302] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e880aa537451b8d60d9d089f07f82, f4f2240aef90489bb774464c943b8237; no voters:
I20260620 21:55:51.953902 25642 raft_consensus.cc:3037] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:51.954054 25722 raft_consensus.cc:2781] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:51.954268 25722 raft_consensus.cc:686] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 1 LEADER]: Becoming Leader. State: Replica: f4f2240aef90489bb774464c943b8237, State: Running, Role: LEADER
I20260620 21:55:51.954514 25722 consensus_queue.cc:237] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.954981 25642 raft_consensus.cc:2454] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f2240aef90489bb774464c943b8237 in term 1.
I20260620 21:55:51.957083 25205 catalog_manager.cc:5495] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f2240aef90489bb774464c943b8237 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "f4f2240aef90489bb774464c943b8237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } health_report { overall_health: UNKNOWN } } }
I20260620 21:55:51.967033 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:51.963161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.17.67.193:46035)
0620 21:55:51.963459 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:51.963469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:51.963484 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:51.963579 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:51.963582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:51.963632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:51.963784 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:51.964014 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.964558 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.965466 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:51.965788 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:51.965898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:51.966164 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:51.966637 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:51.966777 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:51.966817 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 21:55:51.966869 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":132,"threads_started":1}
I20260620 21:55:51.969735 25502 raft_consensus.cc:1261] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Refusing update from remote peer f4f2240aef90489bb774464c943b8237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.969751 25642 raft_consensus.cc:1261] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Refusing update from remote peer f4f2240aef90489bb774464c943b8237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:51.970219 25722 consensus_queue.cc:1035] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.970400 25800 consensus_queue.cc:1035] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:51.981917 25205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.982137 25205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:51.984545 25205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Sending DeleteTablet for 3 replicas of tablet f1f01a5016dc4742b6f7e25396e3d166
I20260620 21:55:51.985191 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:48382
I20260620 21:55:51.985347 25342 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 21:55:51.985404 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f01a5016dc4742b6f7e25396e3d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:51 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:36074
I20260620 21:55:51.985427 25779 tablet_replica.cc:330] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: stopping tablet replica
I20260620 21:55:51.985505 25781 tablet_replica.cc:330] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: stopping tablet replica
I20260620 21:55:51.985599 25780 tablet_replica.cc:330] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: stopping tablet replica
I20260620 21:55:51.985589 25779 raft_consensus.cc:2229] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:51.985626 25781 raft_consensus.cc:2229] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:51.985718 25779 raft_consensus.cc:2258] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.985719 25780 raft_consensus.cc:2229] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:51.985849 25780 raft_consensus.cc:2258] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.985987 25781 raft_consensus.cc:2258] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:51.986560 25205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:55:51.986729 25781 ts_tablet_manager.cc:1905] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:55:51.986955 25205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:55:51.989447 25781 ts_tablet_manager.cc:1918] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.989523 25781 log.cc:1196] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/f1f01a5016dc4742b6f7e25396e3d166
I20260620 21:55:51.989827 25781 ts_tablet_manager.cc:1939] T f1f01a5016dc4742b6f7e25396e3d166 P f4f2240aef90489bb774464c943b8237: Deleting consensus metadata
I20260620 21:55:51.990932 25779 ts_tablet_manager.cc:1905] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.991297 25190 catalog_manager.cc:4841] TS f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035): tablet f1f01a5016dc4742b6f7e25396e3d166 (table test-table [id=5872aebc15b246b1a1da929ebf283a5d]) successfully deleted
I20260620 21:55:51.991618 25482 tablet_service.cc:1467] Processing CreateTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 (DEFAULT_TABLE table=test-table [id=7d19e95681d14f549efff88b03bc684e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.991887 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bf3d2b0d924690b9c57ecc38b016b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.993320 25779 ts_tablet_manager.cc:1918] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.993399 25779 log.cc:1196] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/f1f01a5016dc4742b6f7e25396e3d166
I20260620 21:55:51.993682 25779 ts_tablet_manager.cc:1939] T f1f01a5016dc4742b6f7e25396e3d166 P f53c1cc5fc11440eb80482a568c43840: Deleting consensus metadata
I20260620 21:55:51.993810 25718 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Bootstrap starting.
I20260620 21:55:51.994105 25342 tablet_service.cc:1467] Processing CreateTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 (DEFAULT_TABLE table=test-table [id=7d19e95681d14f549efff88b03bc684e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.994323 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bf3d2b0d924690b9c57ecc38b016b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.994606 25718 tablet_bootstrap.cc:654] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.994978 25780 ts_tablet_manager.cc:1905] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:51.995183 25622 tablet_service.cc:1467] Processing CreateTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 (DEFAULT_TABLE table=test-table [id=7d19e95681d14f549efff88b03bc684e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:55:51.995404 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bf3d2b0d924690b9c57ecc38b016b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:51.997074 25780 ts_tablet_manager.cc:1918] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:55:51.997162 25780 log.cc:1196] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/f1f01a5016dc4742b6f7e25396e3d166
I20260620 21:55:51.997437 25780 ts_tablet_manager.cc:1939] T f1f01a5016dc4742b6f7e25396e3d166 P 157e880aa537451b8d60d9d089f07f82: Deleting consensus metadata
I20260620 21:55:51.997440 25716 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Bootstrap starting.
I20260620 21:55:51.997705 25717 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: Bootstrap starting.
I20260620 21:55:51.998073 25190 catalog_manager.cc:4841] TS f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517): tablet f1f01a5016dc4742b6f7e25396e3d166 (table test-table [id=5872aebc15b246b1a1da929ebf283a5d]) successfully deleted
I20260620 21:55:51.998256 25716 tablet_bootstrap.cc:654] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.998534 25717 tablet_bootstrap.cc:654] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:51.999357 25718 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: No bootstrap required, opened a new log
I20260620 21:55:51.999464 25718 ts_tablet_manager.cc:1397] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 21:55:51.999591 25191 catalog_manager.cc:4841] TS 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565): tablet f1f01a5016dc4742b6f7e25396e3d166 (table test-table [id=5872aebc15b246b1a1da929ebf283a5d]) successfully deleted
I20260620 21:55:51.999874 25718 raft_consensus.cc:348] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:51.999991 25718 raft_consensus.cc:374] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:52.000036 25718 raft_consensus.cc:729] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e880aa537451b8d60d9d089f07f82, State: Initialized, Role: FOLLOWER
I20260620 21:55:52.000157 25718 consensus_queue.cc:260] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.000231 25717 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: No bootstrap required, opened a new log
I20260620 21:55:52.000298 25717 ts_tablet_manager.cc:1397] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 21:55:52.000448 25718 ts_tablet_manager.cc:1428] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:52.000514 25716 tablet_bootstrap.cc:492] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: No bootstrap required, opened a new log
I20260620 21:55:52.000584 25716 ts_tablet_manager.cc:1397] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:52.000717 25717 raft_consensus.cc:348] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.000818 25717 raft_consensus.cc:374] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:52.000849 25717 raft_consensus.cc:729] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f2240aef90489bb774464c943b8237, State: Initialized, Role: FOLLOWER
I20260620 21:55:52.000934 25716 raft_consensus.cc:348] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.000975 25717 consensus_queue.cc:260] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.001036 25716 raft_consensus.cc:374] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:52.001066 25716 raft_consensus.cc:729] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Initialized, Role: FOLLOWER
I20260620 21:55:52.001184 25716 consensus_queue.cc:260] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.001214 25717 ts_tablet_manager.cc:1428] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:55:52.001411 25716 ts_tablet_manager.cc:1428] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:55:52.289192 25776 raft_consensus.cc:482] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:55:52.289358 25776 raft_consensus.cc:504] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.289865 25776 leader_election.cc:288] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:52.290493 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bf3d2b0d924690b9c57ecc38b016b9" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237" is_pre_election: true
I20260620 21:55:52.290526 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bf3d2b0d924690b9c57ecc38b016b9" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82" is_pre_election: true
I20260620 21:55:52.290657 25362 raft_consensus.cc:2454] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:52.290714 25502 raft_consensus.cc:2454] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 0.
I20260620 21:55:52.290987 25574 leader_election.cc:302] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:52.291188 25776 raft_consensus.cc:2781] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:55:52.291258 25776 raft_consensus.cc:482] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:55:52.291298 25776 raft_consensus.cc:3037] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:52.292282 25776 raft_consensus.cc:504] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.292599 25776 leader_election.cc:288] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Requested vote from peers f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035), 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565)
I20260620 21:55:52.292974 25362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bf3d2b0d924690b9c57ecc38b016b9" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f2240aef90489bb774464c943b8237"
I20260620 21:55:52.293036 25502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bf3d2b0d924690b9c57ecc38b016b9" candidate_uuid: "f53c1cc5fc11440eb80482a568c43840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157e880aa537451b8d60d9d089f07f82"
I20260620 21:55:52.293102 25362 raft_consensus.cc:3037] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:52.293190 25502 raft_consensus.cc:3037] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:52.294047 25362 raft_consensus.cc:2454] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:52.294250 25502 raft_consensus.cc:2454] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53c1cc5fc11440eb80482a568c43840 in term 1.
I20260620 21:55:52.294317 25574 leader_election.cc:302] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4f2240aef90489bb774464c943b8237, f53c1cc5fc11440eb80482a568c43840; no voters:
I20260620 21:55:52.294485 25776 raft_consensus.cc:2781] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:52.294696 25776 raft_consensus.cc:686] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 1 LEADER]: Becoming Leader. State: Replica: f53c1cc5fc11440eb80482a568c43840, State: Running, Role: LEADER
I20260620 21:55:52.294872 25776 consensus_queue.cc:237] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } }
I20260620 21:55:52.296444 25204 catalog_manager.cc:5495] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53c1cc5fc11440eb80482a568c43840 (127.17.67.195). New cstate: current_term: 1 leader_uuid: "f53c1cc5fc11440eb80482a568c43840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53c1cc5fc11440eb80482a568c43840" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 34517 } health_report { overall_health: HEALTHY } } }
I20260620 21:55:52.374879 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:52.371724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.17.67.195:34517)
0620 21:55:52.371839 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:52.371843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:52.371922 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:55:52.372073 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:52.372076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:52.372127 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:52.372209 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:52.372458 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:52.372964 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:52.373869 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:52.374084 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:52.374192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:52.374433 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:52.374500 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:52.374577 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:52.374668 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 21:55:52.374733 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:55:52.381259 25502 raft_consensus.cc:1261] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Refusing update from remote peer f53c1cc5fc11440eb80482a568c43840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:52.381268 25362 raft_consensus.cc:1261] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Refusing update from remote peer f53c1cc5fc11440eb80482a568c43840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:52.381676 25812 consensus_queue.cc:1035] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157e880aa537451b8d60d9d089f07f82" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:52.381840 25776 consensus_queue.cc:1035] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f2240aef90489bb774464c943b8237" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:52.412470 25205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:52.412626 25205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:55:52.414562 25205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 546520af25be423f8296d1b4ebf3483e: Sending DeleteTablet for 3 replicas of tablet 87bf3d2b0d924690b9c57ecc38b016b9
I20260620 21:55:52.415112 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:52 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:48382
I20260620 21:55:52.415133 25342 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:52 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 21:55:52.415274 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bf3d2b0d924690b9c57ecc38b016b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:55:52 UTC) from {username='oryx', principal='oryx/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:36074
I20260620 21:55:52.415309 25779 tablet_replica.cc:330] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: stopping tablet replica
I20260620 21:55:52.415350 25781 tablet_replica.cc:330] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237: stopping tablet replica
I20260620 21:55:52.415429 25780 tablet_replica.cc:330] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: stopping tablet replica
I20260620 21:55:52.415429 25779 raft_consensus.cc:2229] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:55:52.415483 25781 raft_consensus.cc:2229] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:52.415537 25780 raft_consensus.cc:2229] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:55:52.415589 25781 raft_consensus.cc:2258] T 87bf3d2b0d924690b9c57ecc38b016b9 P f4f2240aef90489bb774464c943b8237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:52.415633 25780 raft_consensus.cc:2258] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:52.415623 25779 raft_consensus.cc:2258] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:55:52.416230 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25277
I20260620 21:55:52.416277 25779 ts_tablet_manager.cc:1905] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:52.416297 25780 ts_tablet_manager.cc:1905] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:55:52.418736 25779 ts_tablet_manager.cc:1918] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:52.418736 25780 ts_tablet_manager.cc:1918] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:55:52.418804 25779 log.cc:1196] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/87bf3d2b0d924690b9c57ecc38b016b9
I20260620 21:55:52.418804 25780 log.cc:1196] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/87bf3d2b0d924690b9c57ecc38b016b9
I20260620 21:55:52.419061 25780 ts_tablet_manager.cc:1939] T 87bf3d2b0d924690b9c57ecc38b016b9 P 157e880aa537451b8d60d9d089f07f82: Deleting consensus metadata
I20260620 21:55:52.419061 25779 ts_tablet_manager.cc:1939] T 87bf3d2b0d924690b9c57ecc38b016b9 P f53c1cc5fc11440eb80482a568c43840: Deleting consensus metadata
I20260620 21:55:52.419874 25190 catalog_manager.cc:4841] TS f53c1cc5fc11440eb80482a568c43840 (127.17.67.195:34517): tablet 87bf3d2b0d924690b9c57ecc38b016b9 (table test-table [id=7d19e95681d14f549efff88b03bc684e]) successfully deleted
I20260620 21:55:52.419965 25191 catalog_manager.cc:4841] TS 157e880aa537451b8d60d9d089f07f82 (127.17.67.194:33565): tablet 87bf3d2b0d924690b9c57ecc38b016b9 (table test-table [id=7d19e95681d14f549efff88b03bc684e]) successfully deleted
W20260620 21:55:52.424880 25190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:46035 (error 108)
I20260620 21:55:52.425055 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25417
I20260620 21:55:52.426661 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:52.426187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:46035 (local address 127.0.0.1:58790)
0620 21:55:52.426433 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:52.426531 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:46035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
W20260620 21:55:52.426872 25190 catalog_manager.cc:4584] TS f4f2240aef90489bb774464c943b8237 (127.17.67.193:46035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87bf3d2b0d924690b9c57ecc38b016b9: Network error: Client connection negotiation failed: client connection to 127.17.67.193:46035: connect: Connection refused (error 111)
I20260620 21:55:52.431936 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25557
I20260620 21:55:52.439240 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25173
2026-06-20T21:55:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3620 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:52 dist-test-slave-cx8d krb5kdc[25822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:52 dist-test-slave-cx8d krb5kdc[25822](info): set up 2 sockets
Jun 20 21:55:52 dist-test-slave-cx8d krb5kdc[25822](info): commencing operation
krb5kdc: starting...
W20260620 21:55:53.486374 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:53 dist-test-slave-cx8d krb5kdc[25822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:53.609714 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45651
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:53.720009 25835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:53.720295 25835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:53.720337 25835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:53.724434 25835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:53.724507 25835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:53.724529 25835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:53.724546 25835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:53.724561 25835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:53.730072 25835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:53.731693 25835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:53.732805 25835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:53.741771 25840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:53.741758 25841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:53.741758 25843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:53.741820 25835 server_base.cc:1034] running on GCE node
I20260620 21:55:53.742682 25835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:53.743858 25835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:53.745018 25835 hybrid_clock.cc:648] HybridClock initialized: now 1781992553745003 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T21:55:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1317 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:53 dist-test-slave-cx8d krb5kdc[25848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:53 dist-test-slave-cx8d krb5kdc[25848](info): set up 2 sockets
Jun 20 21:55:53 dist-test-slave-cx8d krb5kdc[25848](info): commencing operation
krb5kdc: starting...
W20260620 21:55:54.818817 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:55:54 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:55:54.938910 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35513
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:55.048538 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:55.048858 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:55.048926 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:55.052970 25861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:55.053069 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:55.053109 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:55.053145 25861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:55.053180 25861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:55.058454 25861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:55.059576 25861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:55.060613 25861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:55.068747 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.068759 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.068779 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:55.069149 25861 server_base.cc:1034] running on GCE node
I20260620 21:55:55.069787 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:55.070845 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:55.072041 25861 hybrid_clock.cc:648] HybridClock initialized: now 1781992555072032 us; error 34 us; skew 500 ppm
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:55.075062 25861 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:55.075943 25861 webserver.cc:460] Webserver started at http://127.17.67.254:40351/ using document root <none> and password file <none>
I20260620 21:55:55.076499 25861 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:55.076591 25861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:55.076821 25861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:55.078914 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "b0c40aa63ce045b29fb6165b53a81f6a"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "fda3decf1857a5974b42893c2c9458b5"
server_key_iv: "376eee6350178a62de901e9d2ce0ba21"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.079490 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "b0c40aa63ce045b29fb6165b53a81f6a"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "fda3decf1857a5974b42893c2c9458b5"
server_key_iv: "376eee6350178a62de901e9d2ce0ba21"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.083245 25861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 21:55:55.085665 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.086866 25861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:55.087002 25861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "b0c40aa63ce045b29fb6165b53a81f6a"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "fda3decf1857a5974b42893c2c9458b5"
server_key_iv: "376eee6350178a62de901e9d2ce0ba21"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.087114 25861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:55.103885 25861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:55.107496 25861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:55.107724 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:55.115216 25861 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:35513
I20260620 21:55:55.115255 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:35513 every 8 connection(s)
I20260620 21:55:55.116235 25861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:55.119097 25929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:55.124439 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Bootstrap starting.
I20260620 21:55:55.125504 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25861
I20260620 21:55:55.125700 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:55:55.125979 17679 external_mini_cluster.cc:1428] Setting key d789f4e5327d8fbd6168a31606be729f
I20260620 21:55:55.127136 25929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:55.127869 25929 log.cc:824] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:55.130043 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: No bootstrap required, opened a new log
I20260620 21:55:55.134431 25929 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:55.134727 25929 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:55.134820 25929 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c40aa63ce045b29fb6165b53a81f6a, State: Initialized, Role: FOLLOWER
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:55.135247 25929 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:55.135423 25929 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:55.135493 25929 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:55.135591 25929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:55.136457 25929 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:55.136833 25929 leader_election.cc:302] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c40aa63ce045b29fb6165b53a81f6a; no voters:
I20260620 21:55:55.137228 25929 leader_election.cc:288] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:55.137363 25934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:55.137629 25934 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 LEADER]: Becoming Leader. State: Replica: b0c40aa63ce045b29fb6165b53a81f6a, State: Running, Role: LEADER
I20260620 21:55:55.138023 25934 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:55.138609 25929 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:55.139974 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c40aa63ce045b29fb6165b53a81f6a. Latest consensus state: current_term: 1 leader_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } } }
I20260620 21:55:55.139974 25935 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } } }
I20260620 21:55:55.140136 25935 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:55.140136 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:55.140623 25945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:55.142108 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.127126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.17.67.254:35513)
0620 21:55:55.127565 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:55.127582 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:55.127618 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:55:55.128299 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:55.128316 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:55.128624 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:55.128898 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:55.129782 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.130838 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.132059 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.132626 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.132771 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.136034 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:55.136048 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:55.136062 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:55.136092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:55.138244 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.138772 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.138778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.138784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.138916 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.139244 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.139247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.139249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.139813 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:55.140742 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:55.140912 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:55:55.141246 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260620 21:55:55.144053 25945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:55.149592 25945 catalog_manager.cc:1347] Generated new cluster ID: 8cacad65a26d41ef9a4f769c8fe6693a
I20260620 21:55:55.149667 25945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:55.175493 25945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:55.177007 25945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:55.189289 25945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Generated new TSK 0
I20260620 21:55:55.190140 25945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:55:55.224489 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:55.360020 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:55.360317 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:55.360365 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:55.364486 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:55.364552 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:55.364617 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:55.370039 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:55.371078 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:55.372061 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:55.381729 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.381729 25960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.381726 25961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:55.382251 25955 server_base.cc:1034] running on GCE node
I20260620 21:55:55.382647 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:55.383342 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:55.384522 25955 hybrid_clock.cc:648] HybridClock initialized: now 1781992555384495 us; error 42 us; skew 500 ppm
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:55.387634 25955 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:55:55.388571 25955 webserver.cc:460] Webserver started at http://127.17.67.193:44825/ using document root <none> and password file <none>
I20260620 21:55:55.389191 25955 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:55.389293 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:55.389545 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:55.391812 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "bcb3de5c62344511ae8765bb67713aac"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "06f6b14228c72c8c970ee77f93efee3e"
server_key_iv: "eee9f36c590237203fe3831b1e617df4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.392438 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "bcb3de5c62344511ae8765bb67713aac"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "06f6b14228c72c8c970ee77f93efee3e"
server_key_iv: "eee9f36c590237203fe3831b1e617df4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.396471 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 21:55:55.399173 25970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.400595 25955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:55.400772 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "bcb3de5c62344511ae8765bb67713aac"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "06f6b14228c72c8c970ee77f93efee3e"
server_key_iv: "eee9f36c590237203fe3831b1e617df4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.400905 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:55.426376 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:55.430027 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:55.430250 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:55.430907 25955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:55.431793 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:55.431869 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.431948 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:55.432003 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.441794 25955 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:36655
I20260620 21:55:55.441843 26083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:36655 every 8 connection(s)
I20260620 21:55:55.442741 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:55.452394 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 25955
I20260620 21:55:55.452540 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:55:55.452852 17679 external_mini_cluster.cc:1428] Setting key 2cdc9b6802ed06a6bd24cd55b9c5c414
I20260620 21:55:55.456630 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.444683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57053 (local address 127.17.67.254:35513)
0620 21:55:55.444836 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:55.444839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:55.445881 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 21:55:55.446805 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:55.446808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:55.446868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:55.446974 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:55.448163 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.448707 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.449921 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.450169 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.450283 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.452949 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:55.452953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:55.452956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:55.452984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:55.454589 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.455102 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.455108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.455110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.455193 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.455557 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.455562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.455563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.455744 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:55.455851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:55.456343 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 21:55:55.456473 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:55:55.457780 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.445013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:57053)
0620 21:55:55.445736 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:55.445778 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:55.446583 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:55.447169 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:55.447176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:55.447537 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:55.448016 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.448030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.448834 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.448840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.449798 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.449808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.450323 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.450327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.450506 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.451238 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:55.451253 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:55.452766 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:55.454705 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.454711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.454726 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.454983 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.455285 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.455289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.455291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.455443 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.455849 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:55.455856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:55.456106 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 21:55:55.456294 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:55:55.456631 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":118,"threads_started":1}
I20260620 21:55:55.459237 26084 heartbeater.cc:347] Connected to a master server at 127.17.67.254:35513
I20260620 21:55:55.459523 26084 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:55.460065 26084 heartbeater.cc:511] Master 127.17.67.254:35513 requested a full tablet report, sending...
I20260620 21:55:55.461755 25893 ts_manager.cc:194] Registered new tserver with Master: bcb3de5c62344511ae8765bb67713aac (127.17.67.193:36655)
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:55:55.463001 25893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:57053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:55:55.480937 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:55.591117 26089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:55.591378 26089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:55.591423 26089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:55.595533 26089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:55.595621 26089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:55.595683 26089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:55.601078 26089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:55.602273 26089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:55.603415 26089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:55.613371 26097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.613389 26094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.613364 26095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:55.613554 26089 server_base.cc:1034] running on GCE node
I20260620 21:55:55.614028 26089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:55.614660 26089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:55.615830 26089 hybrid_clock.cc:648] HybridClock initialized: now 1781992555615811 us; error 34 us; skew 500 ppm
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:55.618711 26089 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:55:55.619630 26089 webserver.cc:460] Webserver started at http://127.17.67.194:43659/ using document root <none> and password file <none>
I20260620 21:55:55.620201 26089 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:55.620288 26089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:55.620508 26089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:55.622586 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "d38d1bada4fb4e75ae4ee37dd0a79dfe"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "1cc92e337d10c63dcedc8f3673dbbfe7"
server_key_iv: "f229a74b79f6f5088b86713651fb2ce4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.623148 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "d38d1bada4fb4e75ae4ee37dd0a79dfe"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "1cc92e337d10c63dcedc8f3673dbbfe7"
server_key_iv: "f229a74b79f6f5088b86713651fb2ce4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.626885 26089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:55:55.629352 26104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.630671 26089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:55:55.630820 26089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "d38d1bada4fb4e75ae4ee37dd0a79dfe"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "1cc92e337d10c63dcedc8f3673dbbfe7"
server_key_iv: "f229a74b79f6f5088b86713651fb2ce4"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.630932 26089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:55.655913 26089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:55.659296 26089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:55.659495 26089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:55.660100 26089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:55.660933 26089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:55.661002 26089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.661082 26089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:55.661129 26089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.670496 26089 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40223
I20260620 21:55:55.670516 26217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40223 every 8 connection(s)
I20260620 21:55:55.671408 26089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:55.677736 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26089
I20260620 21:55:55.677834 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:55:55.678120 17679 external_mini_cluster.cc:1428] Setting key 36e30419573aec17e4f6a51c59f195cd
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:55.685918 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.673229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:55071 (local address 127.17.67.254:35513)
0620 21:55:55.673377 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:55.673381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:55.674471 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 21:55:55.675341 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:55.675345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:55.675397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:55.675474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:55.676684 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.677220 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.678574 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.678829 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.678946 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.681721 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:55.681726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:55.681731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:55.681763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:55.683499 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.684123 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.684129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.684132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.684232 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.684644 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.684650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.684655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.684931 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:55.685051 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:55.685590 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 21:55:55.685731 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:55:55.686733 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.673663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:55071)
0620 21:55:55.674324 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:55.674365 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:55.675143 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:55.675645 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:55.675654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:55.676015 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:55.676509 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.676523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.677349 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.677352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.678467 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.678477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.678981 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.678993 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.679166 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.679818 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:55.679835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:55.681511 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:55.683630 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.683635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.683648 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.684000 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.684349 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.684352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.684353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.684498 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.685063 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:55.685068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:55.685301 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 21:55:55.685463 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:55:55.685822 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:55:55.687956 26218 heartbeater.cc:347] Connected to a master server at 127.17.67.254:35513
I20260620 21:55:55.688241 26218 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:55.688735 26218 heartbeater.cc:511] Master 127.17.67.254:35513 requested a full tablet report, sending...
I20260620 21:55:55.689937 25893 ts_manager.cc:194] Registered new tserver with Master: d38d1bada4fb4e75ae4ee37dd0a79dfe (127.17.67.194:40223)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:55:55.690775 25893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:55071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:55:55.708748 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:55.820672 26223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:55.820940 26223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:55.820984 26223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:55.825006 26223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:55.825068 26223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:55.825127 26223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:55.830435 26223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:55.831463 26223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:55.832445 26223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:55.842399 26229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.842480 26231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:55.842399 26228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:55.842639 26223 server_base.cc:1034] running on GCE node
I20260620 21:55:55.843130 26223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:55.843868 26223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:55.845073 26223 hybrid_clock.cc:648] HybridClock initialized: now 1781992555845060 us; error 30 us; skew 500 ppm
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:55.847978 26223 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:55:55.848886 26223 webserver.cc:460] Webserver started at http://127.17.67.195:34083/ using document root <none> and password file <none>
I20260620 21:55:55.849452 26223 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:55.849547 26223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:55.849786 26223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:55.851866 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "d8c7d51cbc47485fb8fe3c4a2d18ad25"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "f94da1e0677daec6569900973cd888e1"
server_key_iv: "c2f5e17f1f7d688e8ed3fc9ec353cf4a"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.852432 26223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c7d51cbc47485fb8fe3c4a2d18ad25"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "f94da1e0677daec6569900973cd888e1"
server_key_iv: "c2f5e17f1f7d688e8ed3fc9ec353cf4a"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.856328 26223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 21:55:55.859090 26238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.860657 26223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:55.860816 26223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "d8c7d51cbc47485fb8fe3c4a2d18ad25"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "f94da1e0677daec6569900973cd888e1"
server_key_iv: "c2f5e17f1f7d688e8ed3fc9ec353cf4a"
server_key_version: "encryptionkey@0"
I20260620 21:55:55.860940 26223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:55.879789 26223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:55.883278 26223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:55.883507 26223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:55.884133 26223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:55.885020 26223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:55.885092 26223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.885162 26223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:55.885214 26223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:55.894462 26223 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:35039
I20260620 21:55:55.894475 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:35039 every 8 connection(s)
I20260620 21:55:55.895368 26223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:55.905668 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26223
Jun 20 21:55:55 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:55:55.905824 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:55:55.906095 17679 external_mini_cluster.cc:1428] Setting key d3678bca4d5784ec7cb32abd16f2a2cb
I20260620 21:55:55.910230 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.897319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:53577 (local address 127.17.67.254:35513)
0620 21:55:55.897525 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:55.897531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:55.899043 (+ 1512us) server_negotiation.cc:407] Connection header received
0620 21:55:55.899979 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:55.899983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:55.900043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:55.900123 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:55.901317 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.901980 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.903156 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.903451 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.903565 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.906437 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:55.906444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:55.906447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:55.906482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:55.908190 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.908814 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.908816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.908818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.908893 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:55.909200 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:55.909202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:55.909203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:55.909372 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:55.909445 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:55.909998 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 21:55:55.910104 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:55:55.911125 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:55.898002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:53577)
0620 21:55:55.898871 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:55.898918 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:55.899763 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:55.900299 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:55.900306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:55.900676 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:55.901166 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.901181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.902108 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.902118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.903031 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:55.903040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:55.903611 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:55.903615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:55.903798 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:55.904455 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:55.904472 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:55.906262 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:55.908346 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.908353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.908372 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.908723 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.908974 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:55.908976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:55.908977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:55.909117 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:55.909454 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:55.909459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:55.909768 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 21:55:55.909932 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:55:55.910194 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":138,"threads_started":1}
I20260620 21:55:55.912320 26352 heartbeater.cc:347] Connected to a master server at 127.17.67.254:35513
I20260620 21:55:55.912580 26352 heartbeater.cc:464] Registering TS with master...
I20260620 21:55:55.913080 26352 heartbeater.cc:511] Master 127.17.67.254:35513 requested a full tablet report, sending...
I20260620 21:55:55.914207 25893 ts_manager.cc:194] Registered new tserver with Master: d8c7d51cbc47485fb8fe3c4a2d18ad25 (127.17.67.195:35039)
I20260620 21:55:55.914803 25893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:53577
I20260620 21:55:55.920228 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
Entry for principal oryx/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab.
WARNING: no policy specified for oryx/127.17.67.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
Entry for principal oryx/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab.
WARNING: no policy specified for oryx/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
Entry for principal oryx/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab.
I20260620 21:55:55.992555 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25955
I20260620 21:55:55.998154 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26089
I20260620 21:55:56.003610 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26223
I20260620 21:55:56.008800 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 25861
I20260620 21:55:56.015005 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35513
--webserver_interface=127.17.67.254
--webserver_port=40351
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:56.123286 26361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:56.123581 26361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:56.123664 26361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:56.127686 26361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:56.127767 26361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:56.127806 26361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:56.127866 26361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:56.127907 26361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:56.133139 26361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=40351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:56.134235 26361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:56.135233 26361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:55:56.143610 26361 server_base.cc:1034] running on GCE node
W20260620 21:55:56.143697 26366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.143687 26367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.143719 26369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:56.144562 26361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:56.145678 26361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:56.146854 26361 hybrid_clock.cc:648] HybridClock initialized: now 1781992556146804 us; error 64 us; skew 500 ppm
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992556, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:56.149737 26361 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:55:56.150754 26361 webserver.cc:460] Webserver started at http://127.17.67.254:40351/ using document root <none> and password file <none>
I20260620 21:55:56.151309 26361 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:56.151381 26361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:56.155259 26361 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:55:56.157437 26376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.158777 26361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:56.158875 26361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "b0c40aa63ce045b29fb6165b53a81f6a"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "fda3decf1857a5974b42893c2c9458b5"
server_key_iv: "376eee6350178a62de901e9d2ce0ba21"
server_key_version: "encryptionkey@0"
I20260620 21:55:56.159440 26361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:56.180137 26361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:56.184212 26361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:56.184412 26361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:56.191893 26361 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:35513
I20260620 21:55:56.192214 26428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:35513 every 8 connection(s)
I20260620 21:55:56.192999 26361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:56.197669 26429 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:55:56.199570 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Bootstrap starting.
I20260620 21:55:56.201689 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26361
I20260620 21:55:56.202239 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:36655
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:35513
--webserver_port=44825
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:56.217980 26429 log.cc:824] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:56.224956 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:55:56.225363 26429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Bootstrap complete.
I20260620 21:55:56.229765 26429 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:56.230168 26429 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c40aa63ce045b29fb6165b53a81f6a, State: Initialized, Role: FOLLOWER
I20260620 21:55:56.230696 26429 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:56.230813 26429 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:56.230904 26429 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:56.231027 26429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:55:56.232965 26429 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:56.233315 26429 leader_election.cc:302] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c40aa63ce045b29fb6165b53a81f6a; no voters:
I20260620 21:55:56.233724 26429 leader_election.cc:288] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:55:56.233860 26433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:55:56.234105 26433 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [term 2 LEADER]: Becoming Leader. State: Replica: b0c40aa63ce045b29fb6165b53a81f6a, State: Running, Role: LEADER
I20260620 21:55:56.234472 26433 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } }
I20260620 21:55:56.235008 26429 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:56.236178 26435 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c40aa63ce045b29fb6165b53a81f6a. Latest consensus state: current_term: 2 leader_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } } }
I20260620 21:55:56.236301 26435 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:56.236526 26434 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c40aa63ce045b29fb6165b53a81f6a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35513 } } }
I20260620 21:55:56.236611 26434 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:56.237615 26442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:56.240824 26442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:56.241473 26442 catalog_manager.cc:1259] Loaded cluster ID: 8cacad65a26d41ef9a4f769c8fe6693a
I20260620 21:55:56.241539 26442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:56.244138 26442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:56.244885 26442 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0c40aa63ce045b29fb6165b53a81f6a: Loaded TSK: 0
I20260620 21:55:56.246340 26442 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:55:56.315989 26431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:56.316280 26431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:56.316354 26431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:56.321099 26431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:56.321205 26431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:56.321309 26431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:55:56.327184 26431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.193.keytab
--principal=oryx/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:36655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=44825
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:56.328280 26431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:56.329344 26431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:56.339637 26456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.340090 26458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.340054 26455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:56.340080 26431 server_base.cc:1034] running on GCE node
I20260620 21:55:56.340670 26431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:56.341331 26431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:56.342537 26431 hybrid_clock.cc:648] HybridClock initialized: now 1781992556342499 us; error 54 us; skew 500 ppm
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992556, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:56.345405 26431 init.cc:377] Logged in from keytab as oryx/127.17.67.193@KRBTEST.COM (short username oryx)
I20260620 21:55:56.346390 26431 webserver.cc:460] Webserver started at http://127.17.67.193:44825/ using document root <none> and password file <none>
I20260620 21:55:56.346979 26431 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:56.347060 26431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:56.351364 26431 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:55:56.353605 26465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.355026 26431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:55:56.355271 26431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "bcb3de5c62344511ae8765bb67713aac"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "06f6b14228c72c8c970ee77f93efee3e"
server_key_iv: "eee9f36c590237203fe3831b1e617df4"
server_key_version: "encryptionkey@0"
I20260620 21:55:56.355854 26431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:56.379230 26431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:56.382895 26431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:56.383121 26431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:56.383777 26431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:56.384764 26431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:56.384845 26431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.384915 26431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:56.384997 26431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.394013 26431 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:36655
I20260620 21:55:56.394086 26578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:36655 every 8 connection(s)
I20260620 21:55:56.395020 26431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:55:56.399302 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26431
I20260620 21:55:56.399801 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:40223
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:35513
--webserver_port=43659
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.409463 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.397312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:48875)
0620 21:55:56.398026 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.398066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.398950 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.400246 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.400254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.400661 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.401122 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.401135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.402508 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.402515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.404014 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.404030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.405023 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.405030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.405275 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.405978 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.406013 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.408710 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":125,"threads_started":1}
W20260620 21:55:56.410043 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.397216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:48875 (local address 127.17.67.254:35513)
0620 21:55:56.397773 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.397798 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.398185 (+ 387us) server_negotiation.cc:407] Connection header received
0620 21:55:56.399370 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.399380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.399736 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.400065 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.401294 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.402368 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.404211 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.404815 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.404975 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.409034 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:48875: BlockingRecv error: recv got EOF from 127.17.67.193:48875 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":122,"threads_started":1}
W20260620 21:55:56.410111 26579 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:35513 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.67.254:35513: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.416553 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.410774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:57439)
0620 21:55:56.410998 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.411011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.411073 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.411320 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.411322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.411491 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.411579 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.411584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.412630 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.412633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.413182 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.413190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.413784 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.413787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.413859 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.414413 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.414427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.416432 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260620 21:55:56.416895 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.410901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57439 (local address 127.17.67.254:35513)
0620 21:55:56.411060 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.411064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.411081 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:55:56.411137 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.411142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.411198 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.411301 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.411694 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.412516 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.413289 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.413657 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.413757 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.416752 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:57439: BlockingRecv error: recv got EOF from 127.17.67.193:57439 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.422469 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.417092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:34595)
0620 21:55:56.417188 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.417203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.417275 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.417710 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.417713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.417873 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.417988 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.418005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.418747 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.418750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.419613 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.419623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.420061 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.420065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.420145 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.420592 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.420606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.422382 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 21:55:56.422660 26579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:55:56.422838 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.417321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:34595 (local address 127.17.67.254:35513)
0620 21:55:56.417449 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.417452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.417465 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:56.417498 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.417500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.417535 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.417680 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.418117 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.418626 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.419743 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.419951 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.420038 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.422536 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:34595: BlockingRecv error: recv got EOF from 127.17.67.193:34595 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 21:55:56.515620 26583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:56.515898 26583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:56.515945 26583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:56.520090 26583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:56.520146 26583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:56.520246 26583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:55:56.525420 26583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.194.keytab
--principal=oryx/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:40223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=43659
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:56.526471 26583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:56.527490 26583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:56.536757 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.536754 26589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.536808 26588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:56.537257 26583 server_base.cc:1034] running on GCE node
I20260620 21:55:56.537724 26583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:56.538324 26583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:56.539541 26583 hybrid_clock.cc:648] HybridClock initialized: now 1781992556539522 us; error 29 us; skew 500 ppm
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992556, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:56.542526 26583 init.cc:377] Logged in from keytab as oryx/127.17.67.194@KRBTEST.COM (short username oryx)
I20260620 21:55:56.543372 26583 webserver.cc:460] Webserver started at http://127.17.67.194:43659/ using document root <none> and password file <none>
I20260620 21:55:56.543919 26583 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:56.544045 26583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:56.548516 26583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:56.550868 26598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.552227 26583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:55:56.552353 26583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "d38d1bada4fb4e75ae4ee37dd0a79dfe"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "1cc92e337d10c63dcedc8f3673dbbfe7"
server_key_iv: "f229a74b79f6f5088b86713651fb2ce4"
server_key_version: "encryptionkey@0"
I20260620 21:55:56.552888 26583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:56.564841 26583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:56.567934 26583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:56.568135 26583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:56.568714 26583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:56.569531 26583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:56.569629 26583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.569721 26583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:56.569741 26583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.578279 26583 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40223
I20260620 21:55:56.578354 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40223 every 8 connection(s)
I20260620 21:55:56.579370 26583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:55:56.587172 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26583
I20260620 21:55:56.587836 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:35039
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:35513
--webserver_port=34083
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:55:56.590729 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.581298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:53515 (local address 127.17.67.254:35513)
0620 21:55:56.581439 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.581443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.582554 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 21:55:56.583494 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.583498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.583559 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.583662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.584824 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.585349 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.586544 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.586765 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.586876 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.590603 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:53515: BlockingRecv error: recv got EOF from 127.17.67.194:53515 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:56.591226 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.581752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:53515)
0620 21:55:56.582405 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.582446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.583280 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.583845 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.583851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.584202 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.584682 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.584695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.585478 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.585481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.586415 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.586425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.586935 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.586940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.587108 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.587772 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.587788 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.590381 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":122,"threads_started":1}
W20260620 21:55:56.591753 26712 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:35513 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.67.254:35513: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.597904 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.592229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:58499)
0620 21:55:56.592340 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.592355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.592430 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.592707 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.592709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.592831 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.592917 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.592921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.594182 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.594184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.594707 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.594714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.595408 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.595411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.595479 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.596066 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.596077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.597816 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:55:56.598181 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.592321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58499 (local address 127.17.67.254:35513)
0620 21:55:56.592452 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.592459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.592476 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:55:56.592527 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.592531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.592588 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.592696 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.593065 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.594067 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.594827 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.595293 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.595427 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.598010 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:58499: BlockingRecv error: recv got EOF from 127.17.67.194:58499 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.604314 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.598570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:40053)
0620 21:55:56.598654 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.598669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.598740 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.599070 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.599073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.599223 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.599289 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.599293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.600457 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.600460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.600962 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.600969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.601625 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.601629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.601701 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.602291 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.602304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.604222 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 21:55:56.604498 26581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.598647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:40053 (local address 127.17.67.254:35513)
0620 21:55:56.598783 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.598789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.598807 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:56.598850 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.598854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.598909 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.599006 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.599402 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.600341 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.601081 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.601453 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.601557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.604380 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:40053: BlockingRecv error: recv got EOF from 127.17.67.194:40053 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 21:55:56.604789 26712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:55:56.721709 26715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:56.722096 26715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:56.722193 26715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:56.727614 26715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:56.727670 26715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:56.727731 26715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:55:56.732991 26715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/krb5kdc/oryx_127.17.67.195.keytab
--principal=oryx/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:35039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=34083
--tserver_master_addrs=127.17.67.254:35513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:56.734122 26715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:56.735105 26715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:56.745023 26721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.745168 26720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:56.745021 26723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:56.745263 26715 server_base.cc:1034] running on GCE node
I20260620 21:55:56.745730 26715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:56.746359 26715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:56.747519 26715 hybrid_clock.cc:648] HybridClock initialized: now 1781992556747488 us; error 43 us; skew 500 ppm
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992556, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:56.750272 26715 init.cc:377] Logged in from keytab as oryx/127.17.67.195@KRBTEST.COM (short username oryx)
I20260620 21:55:56.751154 26715 webserver.cc:460] Webserver started at http://127.17.67.195:34083/ using document root <none> and password file <none>
I20260620 21:55:56.751679 26715 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:56.751747 26715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:56.755505 26715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:56.757745 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.758994 26715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:55:56.759085 26715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "d8c7d51cbc47485fb8fe3c4a2d18ad25"
format_stamp: "Formatted at 2026-06-20 21:55:55 on dist-test-slave-cx8d"
server_key: "f94da1e0677daec6569900973cd888e1"
server_key_iv: "c2f5e17f1f7d688e8ed3fc9ec353cf4a"
server_key_version: "encryptionkey@0"
I20260620 21:55:56.759635 26715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:56.770259 26715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:56.772910 26715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:56.773092 26715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:56.773695 26715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:55:56.774468 26715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:55:56.774513 26715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.774578 26715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:55:56.774616 26715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:56.783633 26715 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:35039
I20260620 21:55:56.783665 26843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:35039 every 8 connection(s)
I20260620 21:55:56.784569 26715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:55:56.785404 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26715
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.795387 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.786183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.17.67.254:35513)
0620 21:55:56.786343 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.786348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.786519 (+ 171us) server_negotiation.cc:407] Connection header received
0620 21:55:56.786612 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.786615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.786664 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.786752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.787559 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.788307 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.789507 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.789799 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.789882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.791413 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:56.791419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:56.791431 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:56.791459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:56.793322 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:56.793833 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:56.793843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:56.793847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:56.793966 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:56.794272 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:56.794276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:56.794278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:56.794603 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:56.794763 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:56.795042 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 21:55:56.795270 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 21:55:56.796635 26848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.788201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:47023 (local address 127.17.67.254:35513)
0620 21:55:56.788436 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.788440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.788451 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:56.788825 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.788828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.788869 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.788949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.790391 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.790891 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.792185 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.792405 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.792509 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.796522 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:47023: BlockingRecv error: recv got EOF from 127.17.67.195:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260620 21:55:56.797164 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.786997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:47023)
0620 21:55:56.787699 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.787735 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.788588 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.789135 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.789144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.789620 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.790246 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.790265 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.791013 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.791017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.792050 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.792059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.792546 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.792550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.792729 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.793235 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.793248 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.796353 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":109,"threads_started":1}
W20260620 21:55:56.797610 26844 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:35513 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.67.254:35513: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.802809 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.798030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:35753)
0620 21:55:56.798169 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.798185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.798266 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.798521 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.798524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.798663 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.798747 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.798752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.799539 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.799542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.800103 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.800112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.800506 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.800508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.800576 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.800978 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.800988 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.802724 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 21:55:56.802977 26848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.798128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:35753 (local address 127.17.67.254:35513)
0620 21:55:56.798251 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.798255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.798270 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:56.798361 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.798363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.798405 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.798483 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.798840 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.799338 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.800210 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.800409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.800484 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.802851 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:35753: BlockingRecv error: recv got EOF from 127.17.67.195:35753 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:56 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:56.807837 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:56.803375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:48105)
0620 21:55:56.803489 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:56.803505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:56.803579 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:56.803797 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:56.803799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:56.803911 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:56.803971 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.803975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.804662 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.804664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.805258 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:56.805267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.805716 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.805718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:56.805771 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.806145 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:56.806154 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:56.807755 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:55:56.808025 26848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:56.803441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48105 (local address 127.17.67.254:35513)
0620 21:55:56.803568 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:56.803571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:56.803582 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:56.803674 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:56.803676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:56.803710 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:56.803770 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:56.804107 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.804564 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.805358 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:56.805618 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:56.805707 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:56.807906 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:48105: BlockingRecv error: recv got EOF from 127.17.67.195:48105 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 21:55:56.808039 26844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:57.429311 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:57.423313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:60633)
0620 21:55:57.423739 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:57.423753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:57.423856 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:57.424081 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:57.424083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:57.424297 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:57.424446 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.424455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.425615 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.425619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.426278 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.426290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.426716 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.426719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.426783 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.427321 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:57.427333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:57.429192 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":91,"threads_started":1}
W20260620 21:55:57.429504 26850 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:57.423479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:60633 (local address 127.17.67.254:35513)
0620 21:55:57.423759 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:57.423762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:57.423809 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:55:57.423924 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:57.423926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:57.423966 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:57.424046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:57.424576 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.425344 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.426387 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.426606 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.426685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.429327 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:60633: BlockingRecv error: recv got EOF from 127.17.67.193:60633 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:57.611377 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:57.605459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:39005)
0620 21:55:57.605770 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:57.605783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:57.605882 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:57.606110 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:57.606112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:57.606283 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:57.606375 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.606380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.607290 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.607294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.607904 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.607913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.608353 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.608356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.608496 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.609144 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:57.609159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:57.611221 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":148,"threads_started":1}
W20260620 21:55:57.611469 26850 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:57.605522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:39005 (local address 127.17.67.254:35513)
0620 21:55:57.605677 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:57.605682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:57.605857 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:55:57.605949 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:57.605951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:57.606003 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:57.606079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:57.606471 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.607185 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.608024 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.608256 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.608341 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.611366 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:39005: BlockingRecv error: recv got EOF from 127.17.67.194:39005 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:57 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:57.814589 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:57.808703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:46995)
0620 21:55:57.808986 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:57.808998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:57.809132 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:57.809408 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:57.809411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:57.809671 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:57.809787 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.809793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.810555 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.810562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.811144 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:57.811152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.811623 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.811627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:57.811733 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.812226 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:57.812237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:57.814429 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
W20260620 21:55:57.814875 26850 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:57.808760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:46995 (local address 127.17.67.254:35513)
0620 21:55:57.808903 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:57.808908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:57.809075 (+ 167us) server_negotiation.cc:407] Connection header received
0620 21:55:57.809242 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:57.809246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:57.809300 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:57.809387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:57.809911 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.810442 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.811265 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:57.811491 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:57.811602 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:57.814615 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:46995: BlockingRecv error: recv got EOF from 127.17.67.195:46995 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:58.436224 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:58.430116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:57735)
0620 21:55:58.430417 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:58.430431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:58.430525 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:58.430780 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:58.430782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:58.430956 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:58.431064 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.431070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.431868 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.431871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.432438 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.432446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.432960 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.432966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.433058 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.433685 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:58.433713 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:58.436028 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
W20260620 21:55:58.436297 26854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:58.430316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57735 (local address 127.17.67.254:35513)
0620 21:55:58.430565 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:58.430568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:58.430581 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:58.430620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:58.430622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:58.430662 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:58.430746 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:58.431185 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.431753 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.432580 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.432812 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.432895 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.436173 (+ 3278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:57735: BlockingRecv error: recv got EOF from 127.17.67.193:57735 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:58.617924 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:58.612286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:35491)
0620 21:55:58.612606 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:58.612626 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:58.612756 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:58.613055 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:58.613058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:58.613265 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:58.613353 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.613359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.614188 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.614194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.614767 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.614774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.615241 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.615245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.615330 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.615891 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:58.615913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:58.617729 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
W20260620 21:55:58.617955 26854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:58.612355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:35491 (local address 127.17.67.254:35513)
0620 21:55:58.612552 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:58.612557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:58.612703 (+ 146us) server_negotiation.cc:407] Connection header received
0620 21:55:58.612865 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:58.612870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:58.612941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:58.613029 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:58.613473 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.614076 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.614887 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.615129 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.615238 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.617847 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:35491: BlockingRecv error: recv got EOF from 127.17.67.194:35491 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:58 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:58.825947 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:58.815433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:45357)
0620 21:55:58.815749 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:58.815763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:58.815865 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:58.816154 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:58.816156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:58.816330 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:58.816427 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.816432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.817177 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.817179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.817820 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:58.817829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.818308 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.818310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:58.818371 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.823749 (+ 5378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:58.823763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:58.825819 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":140,"threads_started":1}
W20260620 21:55:58.826102 26854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:58.815561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:45357 (local address 127.17.67.254:35513)
0620 21:55:58.815696 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:58.815699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:58.815829 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:55:58.815941 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:58.815943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:58.815985 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:58.816059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:58.816573 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.817076 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.817957 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:58.818193 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:58.818294 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:58.825949 (+ 7655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:45357: BlockingRecv error: recv got EOF from 127.17.67.195:45357 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:59.442652 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:59.436941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:33285)
0620 21:55:59.437376 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:59.437390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:59.437493 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:59.437761 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:59.437763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:59.437956 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:59.438053 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.438059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.438818 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.438820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.439383 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.439392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.439819 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.439821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.439885 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.440452 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:59.440466 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:59.442529 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":85,"threads_started":1}
W20260620 21:55:59.442857 26858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:59.437078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:33285 (local address 127.17.67.254:35513)
0620 21:55:59.437305 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:59.437309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:59.437460 (+ 151us) server_negotiation.cc:407] Connection header received
0620 21:55:59.437589 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:59.437591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:59.437640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:59.437718 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:59.438185 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.438710 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.439494 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.439720 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.439803 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.442678 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:33285: BlockingRecv error: recv got EOF from 127.17.67.193:33285 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":57,"threads_started":1}
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:59.625294 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:59.618911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:35943)
0620 21:55:59.619178 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:59.619192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:59.619298 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:59.619636 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:59.619638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:59.619814 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:59.619908 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.619913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.620856 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.620860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.621627 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.621644 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.622289 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.622293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.622390 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.622998 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:59.623017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:59.625114 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260620 21:55:59.625365 26858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:59.619009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:35943 (local address 127.17.67.254:35513)
0620 21:55:59.619173 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:59.619177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:59.619251 (+ 74us) server_negotiation.cc:407] Connection header received
0620 21:55:59.619422 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:59.619425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:59.619512 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:59.619603 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:59.620037 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.620695 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.621771 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.622120 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.622228 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.625253 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:35943: BlockingRecv error: recv got EOF from 127.17.67.194:35943 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:55:59 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:55:59.832129 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:59.826716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:47261)
0620 21:55:59.827019 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:59.827032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:59.827156 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:59.827403 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:59.827406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:59.827621 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:59.827710 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.827715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.828446 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.828452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.829031 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:59.829039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.829466 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.829470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:59.829552 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.830098 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:59.830109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:59.831940 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260620 21:55:59.832158 26858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:59.826806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:47261 (local address 127.17.67.254:35513)
0620 21:55:59.826959 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:59.826963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:59.827101 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:55:59.827248 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:59.827252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:59.827302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:59.827384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:59.827821 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.828342 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.829145 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:59.829362 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:59.829473 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:59.832058 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:47261: BlockingRecv error: recv got EOF from 127.17.67.195:47261 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:00.448625 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:00.443386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:38493)
0620 21:56:00.443649 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:00.443662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:00.443750 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:00.443973 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:00.443975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:00.444147 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:00.444240 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.444245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.444962 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.444964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.445507 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.445515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.445984 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.445986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.446048 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.446579 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:00.446591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:00.448511 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
W20260620 21:56:00.448819 26862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:00.443463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:38493 (local address 127.17.67.254:35513)
0620 21:56:00.443709 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:00.443713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:00.443725 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:56:00.443818 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:00.443820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:00.443860 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:00.443937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:00.444332 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.444873 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.445668 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.445887 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.445968 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.448659 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:38493: BlockingRecv error: recv got EOF from 127.17.67.193:38493 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:00.632164 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:00.626163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:38195)
0620 21:56:00.626449 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:00.626464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:00.626602 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:00.626879 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:00.626882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:00.627094 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:00.627189 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.627194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.627971 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.627978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.628541 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.628549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.628977 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.628981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.629067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.629551 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:00.629596 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:00.632024 (+ 2428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
W20260620 21:56:00.632372 26862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:00.626212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:38195 (local address 127.17.67.254:35513)
0620 21:56:00.626416 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:00.626420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:00.626547 (+ 127us) server_negotiation.cc:407] Connection header received
0620 21:56:00.626707 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:00.626711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:00.626762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:00.626848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:00.627305 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.627860 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.628654 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.628872 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.628976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.632163 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:38195: BlockingRecv error: recv got EOF from 127.17.67.194:38195 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:00 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:00.838444 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:00.832947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:58877)
0620 21:56:00.833245 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:00.833259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:00.833359 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:00.833634 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:00.833636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:00.833811 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:00.833903 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.833908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.834594 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.834596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.835132 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:00.835139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.835607 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.835610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:00.835675 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.836236 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:00.836249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:00.838309 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
W20260620 21:56:00.838616 26862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:00.833060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:58877 (local address 127.17.67.254:35513)
0620 21:56:00.833216 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:00.833219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:00.833329 (+ 110us) server_negotiation.cc:407] Connection header received
0620 21:56:00.833434 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:00.833436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:00.833482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:00.833550 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:00.834007 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.834507 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.835228 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:00.835508 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:00.835589 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:00.838463 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:58877: BlockingRecv error: recv got EOF from 127.17.67.195:58877 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:01.454926 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:01.449439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:37317)
0620 21:56:01.449750 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:01.449763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:01.449845 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:01.450052 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:01.450054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:01.450231 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:01.450319 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.450324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.451101 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.451103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.451681 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.451689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.452113 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.452115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.452173 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.452733 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:01.452744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:01.454797 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":154,"threads_started":1}
W20260620 21:56:01.455158 26866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:01.449503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:37317 (local address 127.17.67.254:35513)
0620 21:56:01.449816 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:01.449820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:01.449834 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:56:01.449914 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:01.449916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:01.449958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:01.450036 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:01.450458 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.450999 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.451789 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.452019 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.452101 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.454986 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:37317: BlockingRecv error: recv got EOF from 127.17.67.193:37317 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":64,"threads_started":1}
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:01.638558 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:01.633016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:35501)
0620 21:56:01.633276 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:01.633290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:01.633374 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:01.633687 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:01.633689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:01.633873 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:01.633967 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.633973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.634698 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.634700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.635269 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.635277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.635805 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.635808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.635895 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.636494 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:01.636506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:01.638443 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
W20260620 21:56:01.638836 26866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:01.633163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:35501 (local address 127.17.67.254:35513)
0620 21:56:01.633324 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:01.633327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:01.633343 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:56:01.633504 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:01.633506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:01.633547 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:01.633656 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:01.634067 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.634596 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.635456 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.635686 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.635797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.638593 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:35501: BlockingRecv error: recv got EOF from 127.17.67.194:35501 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:01 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:01.844542 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:01.839225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:40685)
0620 21:56:01.839497 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:01.839512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:01.839616 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:01.839838 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:01.839840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:01.840036 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:01.840145 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.840150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.840882 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.840885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.841468 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:01.841478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.841957 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.841959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:01.842038 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.842512 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:01.842523 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:01.844371 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
W20260620 21:56:01.844604 26866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:01.839329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:40685 (local address 127.17.67.254:35513)
0620 21:56:01.839466 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:01.839470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:01.839593 (+ 123us) server_negotiation.cc:407] Connection header received
0620 21:56:01.839691 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:01.839693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:01.839736 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:01.839804 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:01.840249 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.840782 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.841630 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:01.841859 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:01.841940 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:01.844513 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:40685: BlockingRecv error: recv got EOF from 127.17.67.195:40685 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:02.461755 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:02.455841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:59819)
0620 21:56:02.456247 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:02.456265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:02.456370 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:02.456719 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:02.456722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:02.456935 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:02.457028 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.457033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.457936 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.457941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.458507 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.458515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.458977 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.458981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.459089 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.459629 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:02.459643 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:02.461558 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
W20260620 21:56:02.461817 26870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:02.456034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59819 (local address 127.17.67.254:35513)
0620 21:56:02.456333 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:02.456338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:02.456351 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:56:02.456490 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:02.456495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:02.456580 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:02.456697 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:02.457155 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.457834 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.458623 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.458867 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.458979 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.461700 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:59819: BlockingRecv error: recv got EOF from 127.17.67.193:59819 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:02.645005 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:02.639334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:51947)
0620 21:56:02.639646 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:02.639659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:02.639805 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:02.640264 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:02.640267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:02.640482 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:02.640570 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.640575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.641324 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.641327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.641972 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.641981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.642420 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.642423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.642499 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.643037 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:02.643049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:02.644829 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
W20260620 21:56:02.645049 26870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:02.639384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:51947 (local address 127.17.67.254:35513)
0620 21:56:02.639596 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:02.639601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:02.639746 (+ 145us) server_negotiation.cc:407] Connection header received
0620 21:56:02.640092 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:02.640096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:02.640155 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:02.640243 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:02.640681 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.641207 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.642088 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.642308 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.642425 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.644952 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:51947: BlockingRecv error: recv got EOF from 127.17.67.194:51947 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:02 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:02.851158 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:02.845432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:48793)
0620 21:56:02.845811 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:02.845824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:02.845933 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:02.846223 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:02.846225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:02.846405 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:02.846518 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.846523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.847304 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.847307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.847906 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:02.847916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.848368 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.848371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:02.848471 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.849041 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:02.849053 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:02.850989 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":223,"threads_started":1}
W20260620 21:56:02.851254 26870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:02.845480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48793 (local address 127.17.67.254:35513)
0620 21:56:02.845737 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:02.845742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:02.845901 (+ 159us) server_negotiation.cc:407] Connection header received
0620 21:56:02.846056 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:02.846059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:02.846111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:02.846191 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:02.846645 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.847168 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.848029 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:02.848260 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:02.848352 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:02.851134 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:48793: BlockingRecv error: recv got EOF from 127.17.67.195:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:03.468551 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:03.462510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:36075)
0620 21:56:03.462867 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:03.462882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:03.462991 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:03.463308 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:03.463311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:03.463507 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:03.463603 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.463608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.464508 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.464513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.465282 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.465295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.465933 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.465936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.466002 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.466551 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:03.466563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:03.468431 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
W20260620 21:56:03.468705 26874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:03.462558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:36075 (local address 127.17.67.254:35513)
0620 21:56:03.462857 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:03.462861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:03.462974 (+ 113us) server_negotiation.cc:407] Connection header received
0620 21:56:03.463101 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:03.463103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:03.463146 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:03.463256 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:03.463747 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.464393 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.465411 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.465822 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.465912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.468567 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:36075: BlockingRecv error: recv got EOF from 127.17.67.193:36075 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":119,"threads_started":1}
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:03.651816 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:03.645833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:42621)
0620 21:56:03.646156 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:03.646169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:03.646272 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:03.646543 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:03.646545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:03.646727 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:03.646820 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.646825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.647596 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.647598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.648143 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.648150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.648615 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.648618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.648678 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.649444 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:03.649458 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:03.651635 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
W20260620 21:56:03.651866 26874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:03.645917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:42621 (local address 127.17.67.254:35513)
0620 21:56:03.646091 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:03.646099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:03.646241 (+ 142us) server_negotiation.cc:407] Connection header received
0620 21:56:03.646363 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:03.646365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:03.646443 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:03.646509 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:03.646921 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.647502 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.648248 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.648519 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.648611 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.651759 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:42621: BlockingRecv error: recv got EOF from 127.17.67.194:42621 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:03 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:03.861515 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:03.851942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:55571)
0620 21:56:03.852259 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:03.852272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:03.852375 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:03.852661 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:03.852664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:03.852858 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:03.852957 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.852963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.853786 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.853789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.854395 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:03.854404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.854872 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.854876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:03.854964 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.858945 (+ 3981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:03.858958 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:03.861318 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
W20260620 21:56:03.861558 26874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:03.852040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:55571 (local address 127.17.67.254:35513)
0620 21:56:03.852168 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:03.852172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:03.852371 (+ 199us) server_negotiation.cc:407] Connection header received
0620 21:56:03.852446 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:03.852448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:03.852498 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:03.852579 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:03.853078 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.853687 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.854506 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:03.854739 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:03.854834 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:03.861453 (+ 6619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:55571: BlockingRecv error: recv got EOF from 127.17.67.195:55571 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:04.475270 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:04.469440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:39641)
0620 21:56:04.469733 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:04.469752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:04.469869 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:04.470211 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:04.470213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:04.470467 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:04.470616 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.470625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.471373 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.471376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.471946 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.471954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.472398 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.472400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.472465 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.473131 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:04.473144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:04.475150 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":147,"threads_started":1}
W20260620 21:56:04.475431 26878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:04.469512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:39641 (local address 127.17.67.254:35513)
0620 21:56:04.469773 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:04.469777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:04.469817 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:56:04.470011 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:04.470014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:04.470066 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:04.470171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:04.470729 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.471275 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.472048 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.472266 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.472345 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.475293 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:39641: BlockingRecv error: recv got EOF from 127.17.67.193:39641 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":112,"threads_started":1}
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:04.658277 26879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:04.652556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:52713)
0620 21:56:04.652831 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:04.652846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:04.652952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:04.653184 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:04.653186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:04.653383 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:04.653483 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.653489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.654408 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.654411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.654990 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.654998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.655462 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.655465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.655542 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.656106 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:04.656119 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:04.658158 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
W20260620 21:56:04.658380 26878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:04.652699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:52713 (local address 127.17.67.254:35513)
0620 21:56:04.652858 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:04.652863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:04.652901 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:56:04.653035 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:04.653037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:04.653080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:04.653148 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:04.653630 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.654296 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.655110 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.655351 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.655460 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.658287 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:52713: BlockingRecv error: recv got EOF from 127.17.67.194:52713 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:04 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:04.867997 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:04.862261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:60465)
0620 21:56:04.862594 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:04.862607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:04.862703 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:04.862949 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:04.862951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:04.863120 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:04.863210 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.863215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.864052 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.864055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.864686 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:04.864695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.865167 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.865170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:04.865230 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.865841 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:04.865860 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:04.867870 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":80,"threads_started":1}
W20260620 21:56:04.868170 26878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:04.862355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:60465 (local address 127.17.67.254:35513)
0620 21:56:04.862490 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:04.862494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:04.862689 (+ 195us) server_negotiation.cc:407] Connection header received
0620 21:56:04.862769 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:04.862771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:04.862818 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:04.862926 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:04.863350 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.863937 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.864792 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:04.865065 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:04.865168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:04.868011 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:60465: BlockingRecv error: recv got EOF from 127.17.67.195:60465 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:05.481706 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:05.476092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:50239)
0620 21:56:05.476397 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:05.476452 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:05.476552 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:05.476806 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:05.476808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:05.477009 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:05.477106 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.477112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.477972 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.477975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.478613 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.478622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.479063 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.479066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.479130 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.479637 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:05.479648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:05.481520 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260620 21:56:05.481805 26882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:05.476239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:50239 (local address 127.17.67.254:35513)
0620 21:56:05.476487 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:05.476491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:05.476505 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:56:05.476643 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:05.476645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:05.476690 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:05.476770 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:05.477220 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.477863 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.478724 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.478962 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.479046 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.481679 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:50239: BlockingRecv error: recv got EOF from 127.17.67.193:50239 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:05.665876 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:05.659191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:49095)
0620 21:56:05.659562 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:05.659579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:05.659714 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:05.660080 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:05.660084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:05.660459 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:05.660605 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.660614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.661722 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.661725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.662404 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.662414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.662943 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.662947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.663019 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.663668 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:05.663685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:05.665735 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
W20260620 21:56:05.666080 26882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:05.659378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:49095 (local address 127.17.67.254:35513)
0620 21:56:05.659587 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:05.659590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:05.659671 (+ 81us) server_negotiation.cc:407] Connection header received
0620 21:56:05.659891 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:05.659897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:05.659955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:05.660022 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:05.660765 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.661602 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.662522 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.662803 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.662917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.665936 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:49095: BlockingRecv error: recv got EOF from 127.17.67.194:49095 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:05 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:05.876956 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:05.868847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:41429)
0620 21:56:05.869144 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:05.869158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:05.869307 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:05.869714 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:05.869718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:05.869973 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:05.870196 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.870209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.871557 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.871564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.872670 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:05.872682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.873374 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.873379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:05.873524 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.874464 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:05.874477 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:05.876829 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
W20260620 21:56:05.877164 26882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:05.868922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:41429 (local address 127.17.67.254:35513)
0620 21:56:05.869185 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:05.869192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:05.869243 (+ 51us) server_negotiation.cc:407] Connection header received
0620 21:56:05.869470 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:05.869477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:05.869549 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:05.869684 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:05.870362 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.871408 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.872811 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:05.873191 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:05.873303 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:05.876979 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:41429: BlockingRecv error: recv got EOF from 127.17.67.195:41429 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:06.488111 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:06.482554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:45489)
0620 21:56:06.482829 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:06.482852 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:06.482941 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:06.483157 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:06.483159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:06.483357 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:06.483462 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.483468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.484246 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.484249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.484853 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.484862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.485293 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.485297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.485363 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.485917 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:06.485929 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:06.487986 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
W20260620 21:56:06.488309 26886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:06.482639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:45489 (local address 127.17.67.254:35513)
0620 21:56:06.482907 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:06.482911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:06.482923 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:56:06.483024 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:06.483026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:06.483068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:06.483142 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:06.483573 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.484122 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.484968 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.485188 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.485266 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.488126 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:45489: BlockingRecv error: recv got EOF from 127.17.67.193:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:06.676671 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:06.666689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:36225)
0620 21:56:06.666988 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:06.667001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:06.667133 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:06.667433 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:06.667436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:06.667641 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:06.667727 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.667732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.668463 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.668469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.669025 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.669032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.669463 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.669466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.669543 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.674397 (+ 4854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:06.674410 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:06.676550 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
W20260620 21:56:06.676818 26886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:06.666753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:36225 (local address 127.17.67.254:35513)
0620 21:56:06.666926 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:06.666931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:06.667071 (+ 140us) server_negotiation.cc:407] Connection header received
0620 21:56:06.667230 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:06.667234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:06.667285 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:06.667379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:06.667836 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.668358 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.669141 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.669358 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.669459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.676691 (+ 7232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:36225: BlockingRecv error: recv got EOF from 127.17.67.194:36225 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:06 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:06.883929 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:06.877882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:49929)
0620 21:56:06.878186 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:06.878200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:06.878284 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:06.878611 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:06.878614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:06.878812 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:06.878915 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.878921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.879718 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.879721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.880283 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:06.880292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.880815 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.880818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:06.880904 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.881472 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:06.881483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:06.883804 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
W20260620 21:56:06.884110 26886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:06.878069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:49929 (local address 127.17.67.254:35513)
0620 21:56:06.878226 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:06.878230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:06.878263 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:56:06.878417 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:06.878420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:06.878468 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:06.878541 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:06.879026 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.879592 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.880418 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:06.880680 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:06.880775 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:06.883946 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:49929: BlockingRecv error: recv got EOF from 127.17.67.195:49929 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:07.495615 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:07.488966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:51883)
0620 21:56:07.489335 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:07.489350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:07.489456 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:07.489789 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:07.489792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:07.489971 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:07.490066 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.490072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.491394 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.491399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.492054 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.492065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.492603 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.492606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.492676 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.493374 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:07.493386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:07.495489 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":199,"threads_started":1}
W20260620 21:56:07.495765 26890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:07.489071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:51883 (local address 127.17.67.254:35513)
0620 21:56:07.489304 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:07.489308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:07.489422 (+ 114us) server_negotiation.cc:407] Connection header received
0620 21:56:07.489534 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:07.489537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:07.489630 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:07.489711 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:07.490422 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.491267 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.492166 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.492449 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.492543 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.495626 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:51883: BlockingRecv error: recv got EOF from 127.17.67.193:51883 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:07.683874 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:07.677764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:53591)
0620 21:56:07.678055 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:07.678067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:07.678205 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:07.678523 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:07.678526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:07.678746 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:07.678876 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.678884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.679731 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.679737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.680379 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.680387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.680849 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.680852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.680939 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.681480 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:07.681494 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:07.683695 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
W20260620 21:56:07.683970 26890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:07.677789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:53591 (local address 127.17.67.254:35513)
0620 21:56:07.677958 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:07.677963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:07.678144 (+ 181us) server_negotiation.cc:407] Connection header received
0620 21:56:07.678312 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:07.678316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:07.678399 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:07.678490 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:07.679003 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.679629 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.680496 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.680737 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.680836 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.683832 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:53591: BlockingRecv error: recv got EOF from 127.17.67.194:53591 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:07 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:07.890347 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:07.884706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:54213)
0620 21:56:07.885039 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:07.885053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:07.885186 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:07.885501 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:07.885504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:07.885739 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:07.885840 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.885846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.886591 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.886597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.887179 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:07.887187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.887648 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.887652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:07.887761 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.888259 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:07.888271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:07.890142 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":100,"threads_started":1}
W20260620 21:56:07.890388 26890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:07.884833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:54213 (local address 127.17.67.254:35513)
0620 21:56:07.884998 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:07.885003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:07.885133 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:56:07.885331 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:07.885335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:07.885387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:07.885487 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:07.885950 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.886480 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.887295 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:07.887532 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:07.887646 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:07.890268 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:54213: BlockingRecv error: recv got EOF from 127.17.67.195:54213 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:08.502132 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:08.496442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:40309)
0620 21:56:08.496862 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:08.496876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:08.496981 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:08.497280 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:08.497283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:08.497466 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:08.497591 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.497597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.498343 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.498345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.498884 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.498891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.499407 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.499410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.499519 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.500084 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:08.500096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:08.502021 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":94,"threads_started":1}
W20260620 21:56:08.502286 26894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:08.496603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:40309 (local address 127.17.67.254:35513)
0620 21:56:08.496892 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:08.496895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:08.496950 (+ 55us) server_negotiation.cc:407] Connection header received
0620 21:56:08.497097 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:08.497100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:08.497149 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:08.497228 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:08.497717 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.498247 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.499027 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.499291 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.499435 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.502156 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:40309: BlockingRecv error: recv got EOF from 127.17.67.193:40309 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:08.690793 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:08.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:58157)
0620 21:56:08.685082 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:08.685095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:08.685195 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:08.685531 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:08.685534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:08.685754 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:08.685856 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.685862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.686868 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.686872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.687565 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.687575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.687992 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.687995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.688057 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.688664 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:08.688676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:08.690648 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
W20260620 21:56:08.690933 26894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:08.684898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58157 (local address 127.17.67.254:35513)
0620 21:56:08.685054 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:08.685058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:08.685183 (+ 125us) server_negotiation.cc:407] Connection header received
0620 21:56:08.685308 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:08.685311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:08.685361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:08.685444 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:08.686058 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.686726 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.687670 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.687895 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.687977 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.690806 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:58157: BlockingRecv error: recv got EOF from 127.17.67.194:58157 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:08 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:08.901328 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:08.891175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:48267)
0620 21:56:08.891457 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:08.891472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:08.891577 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:08.891891 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:08.891914 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:08.892103 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:08.892203 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.892209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.892966 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.892969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.893604 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:08.893612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.894038 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.894042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:08.894109 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.899168 (+ 5059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:08.899182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:08.901145 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
W20260620 21:56:08.901387 26894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:08.891282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48267 (local address 127.17.67.254:35513)
0620 21:56:08.891450 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:08.891454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:08.891542 (+ 88us) server_negotiation.cc:407] Connection header received
0620 21:56:08.891696 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:08.891698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:08.891748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:08.891844 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:08.892324 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.892851 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.893701 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:08.893922 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:08.894001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:08.901279 (+ 7278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:48267: BlockingRecv error: recv got EOF from 127.17.67.195:48267 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:09.508108 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:09.502921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:40529)
0620 21:56:09.503173 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:09.503186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:09.503278 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:09.503542 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:09.503544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:09.503722 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:09.503812 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.503817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.504540 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.504543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.505099 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.505106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.505556 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.505559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.505693 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.506168 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:09.506178 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:09.507983 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260620 21:56:09.508294 26898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:09.502962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:40529 (local address 127.17.67.254:35513)
0620 21:56:09.503190 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:09.503193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:09.503232 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:56:09.503390 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:09.503392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:09.503434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:09.503503 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:09.503909 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.504446 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.505200 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.505447 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.505524 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.508132 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:40529: BlockingRecv error: recv got EOF from 127.17.67.193:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:09.697657 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:09.691898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:35481)
0620 21:56:09.692346 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:09.692364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:09.692497 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:09.692774 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:09.692778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:09.692984 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:09.693072 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.693078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.693889 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.693893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.694449 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.694456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.694895 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.694899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.694983 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.695481 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:09.695494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:09.697442 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":200,"threads_started":1}
W20260620 21:56:09.697724 26898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:09.692181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:35481 (local address 127.17.67.254:35513)
0620 21:56:09.692373 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:09.692379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:09.692450 (+ 71us) server_negotiation.cc:407] Connection header received
0620 21:56:09.692598 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:09.692603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:09.692656 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:09.692745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:09.693189 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.693782 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.694565 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.694785 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.694893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.697606 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:35481: BlockingRecv error: recv got EOF from 127.17.67.194:35481 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:09 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:09.908123 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:09.902178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:52303)
0620 21:56:09.902532 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:09.902546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:09.902648 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:09.902955 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:09.902958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:09.903178 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:09.903288 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.903294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.904152 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.904158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.904775 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:09.904791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.905299 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.905302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:09.905401 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.905973 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:09.905985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:09.908003 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":169,"threads_started":1}
W20260620 21:56:09.908314 26898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:09.902253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:52303 (local address 127.17.67.254:35513)
0620 21:56:09.902379 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:09.902386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:09.902625 (+ 239us) server_negotiation.cc:407] Connection header received
0620 21:56:09.902764 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:09.902768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:09.902821 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:09.902927 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:09.903426 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.904031 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.904910 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:09.905178 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:09.905297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:09.908142 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:52303: BlockingRecv error: recv got EOF from 127.17.67.195:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:10.514942 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:10.508976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:36997)
0620 21:56:10.509282 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:10.509296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:10.509392 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:10.509743 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:10.509745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:10.509926 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:10.510022 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.510027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.510913 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.510918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.511775 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.511786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.512216 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.512218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.512340 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.512850 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:10.512861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:10.514806 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
W20260620 21:56:10.515066 26902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:10.509103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:36997 (local address 127.17.67.254:35513)
0620 21:56:10.509453 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:10.509457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:10.509469 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:56:10.509503 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:10.509505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:10.509545 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:10.509658 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:10.510135 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.510800 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.511898 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.512121 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.512202 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.514942 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:36997: BlockingRecv error: recv got EOF from 127.17.67.193:36997 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:10.704531 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:10.698735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:55647)
0620 21:56:10.699013 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:10.699027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:10.699126 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:10.699389 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:10.699393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:10.699579 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:10.699680 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.699686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.700516 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.700519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.701109 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.701118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.701686 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.701689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.701788 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.702410 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:10.702422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:10.704404 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260620 21:56:10.704735 26902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:10.698791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:55647 (local address 127.17.67.254:35513)
0620 21:56:10.698916 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:10.698922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:10.699084 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:56:10.699199 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:10.699201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:10.699243 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:10.699312 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:10.699821 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.700400 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.701231 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.701490 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.701599 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.704561 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:55647: BlockingRecv error: recv got EOF from 127.17.67.194:55647 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:10 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.195@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:10.914413 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:10.908864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.195:44323)
0620 21:56:10.909156 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:10.909170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:10.909351 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:10.909640 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:10.909643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:10.909860 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:10.909945 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.909951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.910706 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.910711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.911280 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:10.911288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.911739 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.911742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:10.911835 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.912352 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:10.912363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:10.914294 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260620 21:56:10.914544 26902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:10.908951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:44323 (local address 127.17.67.254:35513)
0620 21:56:10.909106 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:10.909111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:10.909255 (+ 144us) server_negotiation.cc:407] Connection header received
0620 21:56:10.909453 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:10.909457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:10.909508 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:10.909618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:10.910065 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.910599 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.911400 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:10.911622 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:10.911727 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:10.914428 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.195:44323: BlockingRecv error: recv got EOF from 127.17.67.195:44323 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 21:56:11 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:11 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.193@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:11.521513 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:11.515853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.193:40363)
0620 21:56:11.516150 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:11.516164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:11.516247 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:11.516493 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:11.516495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:11.516671 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:11.516757 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:11.516762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.517539 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.517541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:11.518150 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:11.518159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.518759 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.518763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:11.518880 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:11.519494 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:11.519508 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:11.521391 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
W20260620 21:56:11.521701 26906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:11.515981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:40363 (local address 127.17.67.254:35513)
0620 21:56:11.516226 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:11.516232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:11.516246 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:56:11.516338 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:11.516340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:11.516382 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:11.516461 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:11.516877 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.517430 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.518363 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.518646 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.518734 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:11.521508 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.193:40363: BlockingRecv error: recv got EOF from 127.17.67.193:40363 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
Jun 20 21:56:11 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:56:11 dist-test-slave-cx8d krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.67.194@KRBTEST.COM for oryx/127.17.67.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:56:11.711644 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:11.705613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:35513 (local address 127.17.67.194:54505)
0620 21:56:11.705927 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:11.705942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:11.706046 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:11.706369 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:11.706372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:11.706565 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:11.706659 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:11.706664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.707613 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.707616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:11.708433 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:11.708443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.708935 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.708940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:11.709046 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:11.709617 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:11.709635 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:11.711525 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.67.254:35513: Server oryx/127.17.67.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
W20260620 21:56:11.711848 26906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:11.705775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54505 (local address 127.17.67.254:35513)
0620 21:56:11.705963 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:11.705967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:11.706001 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:56:11.706139 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:11.706141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:11.706186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:11.706269 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:11.706800 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.707461 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.708593 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:11.708812 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:11.708940 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:11.711700 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.67.194:54505: BlockingRecv error: recv got EOF from 127.17.67.194:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:56:11.789947 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26431
I20260620 21:56:11.795707 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26583
I20260620 21:56:11.801036 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26715
I20260620 21:56:11.806856 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26361
2026-06-20T21:56:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18049 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T21:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:56:11Z Disabled control of system clock
I20260620 21:56:11.837967 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36979
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:36979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:56:11.951668 26911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:11.951951 26911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:11.951985 26911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:11.956022 26911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:56:11.956090 26911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:11.956106 26911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:11.956125 26911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:56:11.956139 26911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:56:11.960937 26911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:36979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:11.962102 26911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:11.963454 26911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:56:11.972383 26911 server_base.cc:1034] running on GCE node
W20260620 21:56:11.972563 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:11.972470 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:11.972708 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:11.973764 26911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:11.974973 26911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:11.976187 26911 hybrid_clock.cc:648] HybridClock initialized: now 1781992571976143 us; error 60 us; skew 500 ppm
I20260620 21:56:11.978243 26911 webserver.cc:460] Webserver started at http://127.17.67.254:45261/ using document root <none> and password file <none>
I20260620 21:56:11.978855 26911 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:11.978950 26911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:11.979180 26911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:11.981256 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "24eb02765ca3426b9ff3627951291fc0"
format_stamp: "Formatted at 2026-06-20 21:56:11 on dist-test-slave-cx8d"
server_key: "f14f3d75c488f014b8a161ae40b23bd0"
server_key_iv: "2dcd4e2a1b4893abe9c150e0f8afc503"
server_key_version: "encryptionkey@0"
I20260620 21:56:11.981851 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "24eb02765ca3426b9ff3627951291fc0"
format_stamp: "Formatted at 2026-06-20 21:56:11 on dist-test-slave-cx8d"
server_key: "f14f3d75c488f014b8a161ae40b23bd0"
server_key_iv: "2dcd4e2a1b4893abe9c150e0f8afc503"
server_key_version: "encryptionkey@0"
I20260620 21:56:11.985555 26911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:56:11.988034 26925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:11.989244 26911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:56:11.989375 26911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "24eb02765ca3426b9ff3627951291fc0"
format_stamp: "Formatted at 2026-06-20 21:56:11 on dist-test-slave-cx8d"
server_key: "f14f3d75c488f014b8a161ae40b23bd0"
server_key_iv: "2dcd4e2a1b4893abe9c150e0f8afc503"
server_key_version: "encryptionkey@0"
I20260620 21:56:11.989486 26911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:12.019625 26911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:12.020380 26911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:12.020584 26911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:12.028146 26911 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:36979
I20260620 21:56:12.028152 26977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:36979 every 8 connection(s)
I20260620 21:56:12.029125 26911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:56:12.032348 26978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:12.033963 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 26911
I20260620 21:56:12.034062 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:56:12.034360 17679 external_mini_cluster.cc:1428] Setting key db65175feea2da3e928b4b846a9811fa
I20260620 21:56:12.039373 26978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0: Bootstrap starting.
I20260620 21:56:12.042001 26978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:12.042793 26978 log.cc:824] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:12.044926 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.035785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.17.67.254:36979)
0620 21:56:12.036478 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.036510 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.036555 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:56:12.037458 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.037483 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.038029 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.038423 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.038882 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.040313 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.041210 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.041924 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.042093 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.042944 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:12.042950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:12.042965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:12.043428 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:12.043481 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:12.043502 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:12.043615 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:12.043769 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 21:56:12.044029 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260620 21:56:12.045009 26978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0: No bootstrap required, opened a new log
I20260620 21:56:12.049005 26978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } }
I20260620 21:56:12.049221 26978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:12.049256 26978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24eb02765ca3426b9ff3627951291fc0, State: Initialized, Role: FOLLOWER
I20260620 21:56:12.049655 26978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } }
I20260620 21:56:12.049821 26978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:56:12.049940 26978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:56:12.050060 26978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:12.051062 26978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } }
I20260620 21:56:12.051397 26978 leader_election.cc:302] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24eb02765ca3426b9ff3627951291fc0; no voters:
I20260620 21:56:12.051700 26978 leader_election.cc:288] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:56:12.051815 26983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:56:12.052078 26983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [term 1 LEADER]: Becoming Leader. State: Replica: 24eb02765ca3426b9ff3627951291fc0, State: Running, Role: LEADER
I20260620 21:56:12.052402 26983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } }
I20260620 21:56:12.052909 26978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:56:12.054057 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24eb02765ca3426b9ff3627951291fc0. Latest consensus state: current_term: 1 leader_uuid: "24eb02765ca3426b9ff3627951291fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } } }
I20260620 21:56:12.054188 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:12.054435 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24eb02765ca3426b9ff3627951291fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24eb02765ca3426b9ff3627951291fc0" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36979 } } }
I20260620 21:56:12.054575 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:12.054613 26992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:56:12.057356 26992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:56:12.062676 26992 catalog_manager.cc:1347] Generated new cluster ID: 1ff908e09ca049c580f41c3c2eaa788b
I20260620 21:56:12.062747 26992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:56:12.082894 26992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:56:12.084160 26992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:56:12.095115 26992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24eb02765ca3426b9ff3627951291fc0: Generated new TSK 0
I20260620 21:56:12.095825 26992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:56:12.109453 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--builtin_ntp_servers=127.17.67.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 21:56:12.219177 27002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:12.219431 27002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:12.219470 27002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:12.223511 27002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:12.223574 27002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:12.223627 27002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:56:12.228305 27002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:12.229404 27002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:12.230827 27002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:12.241081 27008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.241065 27007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.241226 27002 server_base.cc:1034] running on GCE node
W20260620 21:56:12.241072 27010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.241901 27002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:12.242579 27002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:12.243775 27002 hybrid_clock.cc:648] HybridClock initialized: now 1781992572243741 us; error 47 us; skew 500 ppm
I20260620 21:56:12.245795 27002 webserver.cc:460] Webserver started at http://127.17.67.193:35743/ using document root <none> and password file <none>
I20260620 21:56:12.246392 27002 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:12.246485 27002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:12.246724 27002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:12.248802 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "c6167f1ff08948a49244eb9726dceb57"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "961a599e0270f568d37be38565885df2"
server_key_iv: "3bd94cc3b073bce7272568bd70a10323"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.249382 27002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "c6167f1ff08948a49244eb9726dceb57"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "961a599e0270f568d37be38565885df2"
server_key_iv: "3bd94cc3b073bce7272568bd70a10323"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.253365 27002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:56:12.256294 27016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.257637 27002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:56:12.257768 27002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "c6167f1ff08948a49244eb9726dceb57"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "961a599e0270f568d37be38565885df2"
server_key_iv: "3bd94cc3b073bce7272568bd70a10323"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.257889 27002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:12.269866 27002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:12.270594 27002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:12.270815 27002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:12.271454 27002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:12.272398 27002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:12.272470 27002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.272542 27002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:12.272585 27002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.281366 27002 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:35011
I20260620 21:56:12.281386 27129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:35011 every 8 connection(s)
I20260620 21:56:12.282320 27002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:56:12.285851 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27002
I20260620 21:56:12.285972 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:56:12.286273 17679 external_mini_cluster.cc:1428] Setting key bc3073b4285adf42f951c9af4fa277d8
I20260620 21:56:12.288167 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--builtin_ntp_servers=127.17.67.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:56:12.292512 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.284129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:44221 (local address 127.17.67.254:36979)
0620 21:56:12.284276 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.284279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.285350 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 21:56:12.287000 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.287004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.287067 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.287171 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.288144 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.288750 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.290417 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.290633 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.290721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.292006 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:12.292010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:12.292012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:12.292025 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:12.292046 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:12.292054 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:12.292130 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:12.292336 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 21:56:12.292393 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:56:12.293985 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.284538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36979 (local address 127.17.67.193:44221)
0620 21:56:12.285169 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:12.285241 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:12.286766 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:12.287350 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:12.287356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:12.287439 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:12.288010 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.288024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.288877 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.288884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.290286 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.290299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.290793 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.290800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.291029 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.291671 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:12.291675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:12.291730 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:12.291817 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:12.292154 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:12.292166 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:12.292173 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:56:12.292326 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 21:56:12.292771 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260620 21:56:12.295328 27130 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36979
I20260620 21:56:12.295581 27130 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:12.296124 27130 heartbeater.cc:511] Master 127.17.67.254:36979 requested a full tablet report, sending...
I20260620 21:56:12.297734 26942 ts_manager.cc:194] Registered new tserver with Master: c6167f1ff08948a49244eb9726dceb57 (127.17.67.193:35011)
I20260620 21:56:12.298867 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.193:44221
W20260620 21:56:12.397528 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:12.397877 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:12.397969 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:12.402565 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:12.402645 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:12.402736 27133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:56:12.407531 27133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:12.408623 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:12.410080 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:12.419427 27139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.419445 27138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.419616 27133 server_base.cc:1034] running on GCE node
W20260620 21:56:12.419452 27141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.420183 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:12.420805 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:12.422011 27133 hybrid_clock.cc:648] HybridClock initialized: now 1781992572421984 us; error 38 us; skew 500 ppm
I20260620 21:56:12.423931 27133 webserver.cc:460] Webserver started at http://127.17.67.194:40311/ using document root <none> and password file <none>
I20260620 21:56:12.424535 27133 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:12.424633 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:12.424875 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:12.426996 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "96d0d8a08374470198b506e6d905c8ab"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "5ae6f13643da96f647bb3f27f0516ef0"
server_key_iv: "a6e8d4587c704034d4558b68744667fd"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.427551 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "96d0d8a08374470198b506e6d905c8ab"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "5ae6f13643da96f647bb3f27f0516ef0"
server_key_iv: "a6e8d4587c704034d4558b68744667fd"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.431250 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:56:12.433789 27147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.435050 27133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:56:12.435194 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "96d0d8a08374470198b506e6d905c8ab"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "5ae6f13643da96f647bb3f27f0516ef0"
server_key_iv: "a6e8d4587c704034d4558b68744667fd"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.435329 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:12.468605 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:12.469476 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:12.469731 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:12.470373 27133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:12.471342 27133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:12.471413 27133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.471486 27133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:12.471547 27133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.480513 27133 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:35597
I20260620 21:56:12.480538 27260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:35597 every 8 connection(s)
I20260620 21:56:12.481454 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:56:12.484970 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27133
I20260620 21:56:12.485066 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:56:12.485402 17679 external_mini_cluster.cc:1428] Setting key 70ccdb1c69f0bcdc6d91150dda7b44da
I20260620 21:56:12.487569 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--builtin_ntp_servers=127.17.67.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:56:12.491767 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.483342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58993 (local address 127.17.67.254:36979)
0620 21:56:12.483490 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.483494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.484518 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 21:56:12.486329 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.486332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.486378 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.486457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.487448 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.488253 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.489449 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.489863 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.489991 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.491098 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:12.491105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:12.491112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:12.491129 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:12.491156 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:12.491168 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:12.491257 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:12.491484 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 21:56:12.491557 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:56:12.492533 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.483713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36979 (local address 127.17.67.194:58993)
0620 21:56:12.484367 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:12.484408 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:12.485981 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:12.486629 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:12.486636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:12.486760 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:12.487288 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.487303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.488394 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.488397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.489294 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.489304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.490016 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.490019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.490162 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.490834 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:12.490836 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:12.490880 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:12.490944 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:12.491284 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:12.491294 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:12.491301 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:56:12.491451 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 21:56:12.491680 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260620 21:56:12.493896 27261 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36979
I20260620 21:56:12.494163 27261 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:12.494680 27261 heartbeater.cc:511] Master 127.17.67.254:36979 requested a full tablet report, sending...
I20260620 21:56:12.495750 26942 ts_manager.cc:194] Registered new tserver with Master: 96d0d8a08374470198b506e6d905c8ab (127.17.67.194:35597)
I20260620 21:56:12.496603 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.194:58993
W20260620 21:56:12.601045 27264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:12.601306 27264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:12.601343 27264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:12.605309 27264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:12.605362 27264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:12.605415 27264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:56:12.610122 27264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:12.611186 27264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:12.612528 27264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:12.621863 27270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.621973 27269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.621870 27272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.622097 27264 server_base.cc:1034] running on GCE node
I20260620 21:56:12.622531 27264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:12.623137 27264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:12.624372 27264 hybrid_clock.cc:648] HybridClock initialized: now 1781992572624335 us; error 47 us; skew 500 ppm
I20260620 21:56:12.626354 27264 webserver.cc:460] Webserver started at http://127.17.67.195:36477/ using document root <none> and password file <none>
I20260620 21:56:12.626950 27264 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:12.627051 27264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:12.627282 27264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:12.629264 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "bc39b6937f3c4fbc8515f8f97398a048"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "f1fe4d275f51b0e8882145a056efc3f0"
server_key_iv: "935731a9b3702d7149e46456d2a759ca"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.629842 27264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "bc39b6937f3c4fbc8515f8f97398a048"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "f1fe4d275f51b0e8882145a056efc3f0"
server_key_iv: "935731a9b3702d7149e46456d2a759ca"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.633297 27264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:56:12.635825 27278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.637089 27264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:56:12.637315 27264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "bc39b6937f3c4fbc8515f8f97398a048"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "f1fe4d275f51b0e8882145a056efc3f0"
server_key_iv: "935731a9b3702d7149e46456d2a759ca"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.637455 27264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:12.653753 27264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:12.654498 27264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:12.654698 27264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:12.655272 27264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:12.656193 27264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:12.656265 27264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.656332 27264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:12.656383 27264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.665431 27264 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:36079
I20260620 21:56:12.665473 27391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:36079 every 8 connection(s)
I20260620 21:56:12.666414 27264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:56:12.674222 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27264
I20260620 21:56:12.674377 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:56:12.674679 17679 external_mini_cluster.cc:1428] Setting key dbd4670d757b9ac2a20b6f8a7cc5e9da
I20260620 21:56:12.675994 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.668422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:47887 (local address 127.17.67.254:36979)
0620 21:56:12.668555 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.668559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.669557 (+ 998us) server_negotiation.cc:407] Connection header received
0620 21:56:12.671030 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.671034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.671094 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.671171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.672078 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.672626 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.673859 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.674125 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.674243 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.675440 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:12.675443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:12.675445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:12.675456 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:12.675497 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:12.675507 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:12.675585 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:12.675776 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 21:56:12.675836 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:56:12.676962 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.668726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36979 (local address 127.17.67.195:47887)
0620 21:56:12.669394 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:12.669436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:12.670781 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:12.671343 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:12.671350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:12.671436 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:12.671923 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.671935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.672755 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.672759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.673726 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:12.673736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:12.674281 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:12.674285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:12.674517 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:12.675165 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:12.675167 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:12.675206 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:12.675268 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:12.675608 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:12.675618 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:12.675624 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:56:12.675759 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 21:56:12.676043 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260620 21:56:12.678169 27392 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36979
I20260620 21:56:12.678375 27392 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:12.678814 27392 heartbeater.cc:511] Master 127.17.67.254:36979 requested a full tablet report, sending...
I20260620 21:56:12.679726 26942 ts_manager.cc:194] Registered new tserver with Master: bc39b6937f3c4fbc8515f8f97398a048 (127.17.67.195:36079)
I20260620 21:56:12.680250 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.195:47887
I20260620 21:56:12.688920 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:56:12.691740 26981 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:12.690934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.17.67.254:36979)
0620 21:56:12.691062 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.691066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.691228 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:56:12.691320 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.691322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.691363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.691432 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.691637 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52588: BlockingRecv error: recv got EOF from 127.0.0.1:52588 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:56:12.692070 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27002
I20260620 21:56:12.697661 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27133
I20260620 21:56:12.703426 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27264
I20260620 21:56:12.708950 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 26911
2026-06-20T21:56:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (900 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T21:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:56:12Z Disabled control of system clock
I20260620 21:56:12.738060 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38209
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:56:12.848925 27403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:12.849184 27403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:12.849221 27403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:12.853190 27403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:56:12.853248 27403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:12.853264 27403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:12.853281 27403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:56:12.853296 27403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:56:12.857975 27403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:38209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:12.858990 27403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:12.860296 27403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:12.868902 27409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.868986 27411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:12.868986 27408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:12.869119 27403 server_base.cc:1034] running on GCE node
I20260620 21:56:12.869783 27403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:12.870841 27403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:12.872051 27403 hybrid_clock.cc:648] HybridClock initialized: now 1781992572872013 us; error 50 us; skew 500 ppm
I20260620 21:56:12.873996 27403 webserver.cc:460] Webserver started at http://127.17.67.254:40639/ using document root <none> and password file <none>
I20260620 21:56:12.874570 27403 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:12.874665 27403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:12.874903 27403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:12.876956 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "48a9a351517440ea88854a3a8aa317b5"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "04f26549dc5ee7897f59572e4fe91bf5"
server_key_iv: "03d3d818a2acd7c361d48e29437c62ce"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.877537 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "48a9a351517440ea88854a3a8aa317b5"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "04f26549dc5ee7897f59572e4fe91bf5"
server_key_iv: "03d3d818a2acd7c361d48e29437c62ce"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.881209 27403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:56:12.883844 27417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:12.885238 27403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:56:12.885396 27403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "48a9a351517440ea88854a3a8aa317b5"
format_stamp: "Formatted at 2026-06-20 21:56:12 on dist-test-slave-cx8d"
server_key: "04f26549dc5ee7897f59572e4fe91bf5"
server_key_iv: "03d3d818a2acd7c361d48e29437c62ce"
server_key_version: "encryptionkey@0"
I20260620 21:56:12.885519 27403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:12.890199 27403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:12.890810 27403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:12.890990 27403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:12.898093 27403 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:38209
I20260620 21:56:12.898101 27469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:38209 every 8 connection(s)
I20260620 21:56:12.899101 27403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:56:12.902443 27470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:12.903782 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27403
I20260620 21:56:12.903930 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:56:12.904318 17679 external_mini_cluster.cc:1428] Setting key 2ed84f63f674cda355737d0465c331df
I20260620 21:56:12.909452 27470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5: Bootstrap starting.
I20260620 21:56:12.910737 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:12.905939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.17.67.254:38209)
0620 21:56:12.906656 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:12.906676 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:12.906720 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:56:12.907591 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:12.907616 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:12.908091 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:12.908475 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:12.909171 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:12.909176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:12.909190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:12.909201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:12.909262 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:12.909283 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:12.909425 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:12.909558 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 21:56:12.909823 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260620 21:56:12.912096 27470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:12.912977 27470 log.cc:824] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:12.914991 27470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5: No bootstrap required, opened a new log
I20260620 21:56:12.918612 27470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } }
I20260620 21:56:12.918856 27470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:12.918937 27470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a9a351517440ea88854a3a8aa317b5, State: Initialized, Role: FOLLOWER
I20260620 21:56:12.919351 27470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } }
I20260620 21:56:12.919507 27470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:56:12.919589 27470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:56:12.919706 27470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:12.920601 27470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } }
I20260620 21:56:12.920943 27470 leader_election.cc:302] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48a9a351517440ea88854a3a8aa317b5; no voters:
I20260620 21:56:12.921242 27470 leader_election.cc:288] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:56:12.921438 27475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:56:12.921993 27475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [term 1 LEADER]: Becoming Leader. State: Replica: 48a9a351517440ea88854a3a8aa317b5, State: Running, Role: LEADER
I20260620 21:56:12.922380 27475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } }
I20260620 21:56:12.922612 27470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:56:12.923897 27477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a9a351517440ea88854a3a8aa317b5. Latest consensus state: current_term: 1 leader_uuid: "48a9a351517440ea88854a3a8aa317b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } } }
I20260620 21:56:12.924018 27477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:12.924355 27476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48a9a351517440ea88854a3a8aa317b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a9a351517440ea88854a3a8aa317b5" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38209 } } }
I20260620 21:56:12.924436 27476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:12.924455 27486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:56:12.927624 27486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:56:12.933715 27486 catalog_manager.cc:1347] Generated new cluster ID: ee87a73c439e4f99a624b7f3b5bf059e
I20260620 21:56:12.933808 27486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:56:12.960894 27486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:56:12.961262 27486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:56:12.969862 27486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48a9a351517440ea88854a3a8aa317b5: Generated new TSK 0
I20260620 21:56:12.970633 27486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:56:12.978348 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--builtin_ntp_servers=127.17.67.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:56:13.101814 27494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:13.102070 27494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:13.102110 27494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:13.106128 27494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:13.106189 27494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:13.106246 27494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:56:13.110903 27494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:13.111907 27494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:13.113270 27494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:13.123088 27500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:13.123095 27499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:13.123174 27494 server_base.cc:1034] running on GCE node
W20260620 21:56:13.123095 27502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:13.123777 27494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:13.124552 27494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:13.125708 27494 hybrid_clock.cc:648] HybridClock initialized: now 1781992573125658 us; error 63 us; skew 500 ppm
I20260620 21:56:13.127545 27494 webserver.cc:460] Webserver started at http://127.17.67.193:43307/ using document root <none> and password file <none>
I20260620 21:56:13.128082 27494 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:13.128144 27494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:13.128306 27494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:13.130419 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "57ba023aea344bd4baac31366f9613e2"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "c4bbcd5e6c0952c756c53f6c65ed4bcc"
server_key_iv: "2a091a81e5228cd50738744ae5e0023b"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.130929 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "57ba023aea344bd4baac31366f9613e2"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "c4bbcd5e6c0952c756c53f6c65ed4bcc"
server_key_iv: "2a091a81e5228cd50738744ae5e0023b"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.134570 27494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:56:13.137128 27508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.138477 27494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:56:13.138603 27494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "57ba023aea344bd4baac31366f9613e2"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "c4bbcd5e6c0952c756c53f6c65ed4bcc"
server_key_iv: "2a091a81e5228cd50738744ae5e0023b"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.138712 27494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:13.143517 27494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:13.144112 27494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:13.144300 27494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:13.144838 27494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:13.145767 27494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:13.145814 27494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.145851 27494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:13.145901 27494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.154686 27494 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:38137
I20260620 21:56:13.154731 27621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:38137 every 8 connection(s)
I20260620 21:56:13.155653 27494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:56:13.161818 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.157581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:46341 (local address 127.17.67.254:38209)
0620 21:56:13.157752 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:13.157764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:13.158930 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 21:56:13.160173 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:13.160176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:13.160213 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:13.160299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:13.161352 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:13.161354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:13.161356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:13.161367 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:13.161386 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:13.161395 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:13.161470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:13.161658 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 21:56:13.161695 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 21:56:13.162716 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.158085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38209 (local address 127.17.67.193:46341)
0620 21:56:13.158776 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:13.158815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:13.159967 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:13.160464 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:13.160471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:13.160556 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:13.161086 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:13.161088 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:13.161127 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:13.161195 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:13.161474 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:13.161480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:13.161484 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:56:13.161691 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:56:13.161918 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260620 21:56:13.163985 27622 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38209
I20260620 21:56:13.164217 27622 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:13.164613 27622 heartbeater.cc:511] Master 127.17.67.254:38209 requested a full tablet report, sending...
I20260620 21:56:13.165302 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27494
I20260620 21:56:13.165426 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:56:13.165789 17679 external_mini_cluster.cc:1428] Setting key ee91e774462378ed7cef15464fc761e6
I20260620 21:56:13.166268 27434 ts_manager.cc:194] Registered new tserver with Master: 57ba023aea344bd4baac31366f9613e2 (127.17.67.193:38137)
I20260620 21:56:13.167972 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--builtin_ntp_servers=127.17.67.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:56:13.281490 27625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:13.281852 27625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:13.281929 27625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:13.286028 27625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:13.286104 27625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:13.286198 27625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:56:13.291224 27625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:13.292330 27625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:13.293814 27625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:13.303459 27630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:13.303467 27633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:13.303469 27631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:13.303566 27625 server_base.cc:1034] running on GCE node
I20260620 21:56:13.304140 27625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:13.304765 27625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:13.305962 27625 hybrid_clock.cc:648] HybridClock initialized: now 1781992573305952 us; error 29 us; skew 500 ppm
I20260620 21:56:13.307734 27625 webserver.cc:460] Webserver started at http://127.17.67.194:32955/ using document root <none> and password file <none>
I20260620 21:56:13.308357 27625 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:13.308449 27625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:13.308676 27625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:13.310810 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "95835c11b5a640ce846334029284e63c"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "fd41baf6d7fa69df59b880ca42e280e5"
server_key_iv: "e38ae11c213b45428ac64de3a985a622"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.311378 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "95835c11b5a640ce846334029284e63c"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "fd41baf6d7fa69df59b880ca42e280e5"
server_key_iv: "e38ae11c213b45428ac64de3a985a622"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.314955 27625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:56:13.317385 27639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.318614 27625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:56:13.318753 27625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "95835c11b5a640ce846334029284e63c"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "fd41baf6d7fa69df59b880ca42e280e5"
server_key_iv: "e38ae11c213b45428ac64de3a985a622"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.318909 27625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:13.323622 27625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:13.324287 27625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:13.324465 27625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:13.325016 27625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:13.325991 27625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:13.326062 27625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.326186 27625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:13.326237 27625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.334983 27625 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:35105
I20260620 21:56:13.335009 27752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:35105 every 8 connection(s)
I20260620 21:56:13.335865 27625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:56:13.342296 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.337801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:33803 (local address 127.17.67.254:38209)
0620 21:56:13.337951 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:13.337954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:13.339001 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 21:56:13.340440 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:13.340444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:13.340495 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:13.340613 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:13.341829 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:13.341833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:13.341836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:13.341851 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:13.341876 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:13.341889 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:13.341998 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:13.342123 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 21:56:13.342162 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:56:13.343124 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.338177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38209 (local address 127.17.67.194:33803)
0620 21:56:13.338836 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:13.338876 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:13.340171 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:13.340783 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:13.340790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:13.340890 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:13.341507 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:13.341509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:13.341548 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:13.341645 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:13.341979 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:13.341985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:13.341990 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:56:13.342122 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 21:56:13.342312 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260620 21:56:13.344115 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27625
I20260620 21:56:13.344246 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:56:13.344298 27753 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38209
I20260620 21:56:13.344498 27753 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:13.344553 17679 external_mini_cluster.cc:1428] Setting key d76b90dcfdd043f57392aae068c8aacf
I20260620 21:56:13.344857 27753 heartbeater.cc:511] Master 127.17.67.254:38209 requested a full tablet report, sending...
I20260620 21:56:13.345928 27434 ts_manager.cc:194] Registered new tserver with Master: 95835c11b5a640ce846334029284e63c (127.17.67.194:35105)
I20260620 21:56:13.346488 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--builtin_ntp_servers=127.17.67.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:56:13.455482 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:13.455767 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:13.455811 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:13.460013 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:13.460083 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:13.460143 27756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:56:13.465049 27756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:13.466208 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:13.467939 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:13.477869 27761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:13.477918 27756 server_base.cc:1034] running on GCE node
W20260620 21:56:13.477859 27762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:13.477882 27764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:13.478492 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:13.479110 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:13.480289 27756 hybrid_clock.cc:648] HybridClock initialized: now 1781992573480276 us; error 27 us; skew 500 ppm
I20260620 21:56:13.482204 27756 webserver.cc:460] Webserver started at http://127.17.67.195:35457/ using document root <none> and password file <none>
I20260620 21:56:13.482767 27756 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:13.482867 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:13.483093 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:13.485087 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "5e21f212709948a49360eb534efad2d6"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "f9ccb3e5d752310db5391a311f174960"
server_key_iv: "afca26dd42db194df37c86638f9a5f74"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.485692 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "5e21f212709948a49360eb534efad2d6"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "f9ccb3e5d752310db5391a311f174960"
server_key_iv: "afca26dd42db194df37c86638f9a5f74"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.489343 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:56:13.492069 27770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.493346 27756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:56:13.493480 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "5e21f212709948a49360eb534efad2d6"
format_stamp: "Formatted at 2026-06-20 21:56:13 on dist-test-slave-cx8d"
server_key: "f9ccb3e5d752310db5391a311f174960"
server_key_iv: "afca26dd42db194df37c86638f9a5f74"
server_key_version: "encryptionkey@0"
I20260620 21:56:13.493614 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:13.498183 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:13.498776 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:13.498965 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:13.499501 27756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:13.500383 27756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:13.500451 27756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.500519 27756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:13.500559 27756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:13.508782 27756 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33079
I20260620 21:56:13.508807 27883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33079 every 8 connection(s)
I20260620 21:56:13.509654 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:56:13.512449 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27756
I20260620 21:56:13.512570 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:56:13.512918 17679 external_mini_cluster.cc:1428] Setting key d3e699cffd781b279f13301b353d634a
I20260620 21:56:13.516270 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.511515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:59609 (local address 127.17.67.254:38209)
0620 21:56:13.511664 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:13.511668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:13.512957 (+ 1289us) server_negotiation.cc:407] Connection header received
0620 21:56:13.514682 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:13.514684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:13.514717 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:13.514773 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:13.515909 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:13.515912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:13.515914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:56:13.515925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:13.515943 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:56:13.515951 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:13.516019 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:13.516141 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 21:56:13.516174 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:56:13.517155 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:13.511899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38209 (local address 127.17.67.195:59609)
0620 21:56:13.512778 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:13.512837 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:13.514468 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:13.514960 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:13.514970 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:13.515085 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:13.515654 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:56:13.515655 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:13.515694 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:56:13.515759 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:13.516026 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:13.516032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:13.516036 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:56:13.516163 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:56:13.516394 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":106,"threads_started":1}
I20260620 21:56:13.518232 27884 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38209
I20260620 21:56:13.518494 27884 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:13.518883 27884 heartbeater.cc:511] Master 127.17.67.254:38209 requested a full tablet report, sending...
I20260620 21:56:13.519836 27434 ts_manager.cc:194] Registered new tserver with Master: 5e21f212709948a49360eb534efad2d6 (127.17.67.195:33079)
I20260620 21:56:13.526796 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:56:13.529537 27473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:56:13.528685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.17.67.254:38209)
0620 21:56:13.528817 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:13.528821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:13.529000 (+ 179us) server_negotiation.cc:407] Connection header received
0620 21:56:13.529097 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:13.529099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:13.529128 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:13.529191 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:13.529414 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35386: BlockingRecv error: recv got EOF from 127.0.0.1:35386 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:56:13.529747 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27494
I20260620 21:56:13.535915 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27625
I20260620 21:56:13.541061 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27756
I20260620 21:56:13.546159 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27403
2026-06-20T21:56:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (837 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:56:13 dist-test-slave-cx8d krb5kdc[27894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:56:13 dist-test-slave-cx8d krb5kdc[27894](info): set up 2 sockets
Jun 20 21:56:13 dist-test-slave-cx8d krb5kdc[27894](info): commencing operation
krb5kdc: starting...
W20260620 21:56:14.674678 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:56:14 dist-test-slave-cx8d krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:56:14.796306 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36865
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:56:14.903594 27907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:14.903869 27907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:14.903911 27907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:14.907946 27907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:56:14.908006 27907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:14.908026 27907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:14.908043 27907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:56:14.908058 27907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:56:14.913393 27907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:14.914505 27907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:14.915534 27907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:14.923900 27912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:14.923918 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:14.923918 27913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:14.924105 27907 server_base.cc:1034] running on GCE node
I20260620 21:56:14.924822 27907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:14.925944 27907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:14.927099 27907 hybrid_clock.cc:648] HybridClock initialized: now 1781992574927084 us; error 28 us; skew 500 ppm
Jun 20 21:56:14 dist-test-slave-cx8d krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992574, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:56:14.934432 27907 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:56:14.935418 27907 webserver.cc:460] Webserver started at http://127.17.67.254:36207/ using document root <none> and password file <none>
I20260620 21:56:14.935981 27907 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:14.936069 27907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:14.936300 27907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:14.938421 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "d3641c62ec0f45fd8973f0634546ba64"
format_stamp: "Formatted at 2026-06-20 21:56:14 on dist-test-slave-cx8d"
server_key: "52dd9f8631a03159bcdf9c0d3f333a7f"
server_key_iv: "794e15d380f7fdd8c8cb0819cd60a78b"
server_key_version: "encryptionkey@0"
I20260620 21:56:14.939003 27907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "d3641c62ec0f45fd8973f0634546ba64"
format_stamp: "Formatted at 2026-06-20 21:56:14 on dist-test-slave-cx8d"
server_key: "52dd9f8631a03159bcdf9c0d3f333a7f"
server_key_iv: "794e15d380f7fdd8c8cb0819cd60a78b"
server_key_version: "encryptionkey@0"
I20260620 21:56:14.942746 27907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:56:14.945127 27922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:14.946408 27907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:56:14.946545 27907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "d3641c62ec0f45fd8973f0634546ba64"
format_stamp: "Formatted at 2026-06-20 21:56:14 on dist-test-slave-cx8d"
server_key: "52dd9f8631a03159bcdf9c0d3f333a7f"
server_key_iv: "794e15d380f7fdd8c8cb0819cd60a78b"
server_key_version: "encryptionkey@0"
I20260620 21:56:14.946655 27907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:14.965060 27907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:14.968253 27907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:14.968497 27907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:14.976046 27907 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:36865
I20260620 21:56:14.976330 27974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:36865 every 8 connection(s)
I20260620 21:56:14.977063 27907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:56:14.979812 27975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:14.982609 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 27907
I20260620 21:56:14.982744 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:56:14.983016 17679 external_mini_cluster.cc:1428] Setting key 78f7b5ac1b8a1b7396f5b62715191055
I20260620 21:56:14.985543 27975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: Bootstrap starting.
I20260620 21:56:14.988076 27975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:14.988940 27975 log.cc:824] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:14.990826 27975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: No bootstrap required, opened a new log
Jun 20 21:56:14 dist-test-slave-cx8d krb5kdc[27894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:56:14.994551 27975 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } }
I20260620 21:56:14.994782 27975 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:14.994838 27975 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3641c62ec0f45fd8973f0634546ba64, State: Initialized, Role: FOLLOWER
I20260620 21:56:14.995309 27975 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } }
I20260620 21:56:14.995491 27975 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:56:14.995558 27975 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:56:14.995654 27975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:14.996519 27975 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } }
I20260620 21:56:14.996855 27975 leader_election.cc:302] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3641c62ec0f45fd8973f0634546ba64; no voters:
I20260620 21:56:14.997198 27975 leader_election.cc:288] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:56:14.997287 27980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:56:14.997473 27980 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [term 1 LEADER]: Becoming Leader. State: Replica: d3641c62ec0f45fd8973f0634546ba64, State: Running, Role: LEADER
I20260620 21:56:14.997850 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:14.984502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.17.67.254:36865)
0620 21:56:14.984897 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:14.984921 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:14.984955 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:56:14.985544 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:14.985559 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:14.986017 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:14.986313 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:14.987245 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:14.988333 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:14.989485 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:14.990010 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:14.990193 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:14.992862 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:14.992868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:14.992880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:56:14.992905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:14.994682 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:14.995170 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:14.995176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:14.995182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:14.995314 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:14.995614 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:14.995619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:14.995621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:14.996049 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:14.996247 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:14.996533 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 21:56:14.996842 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":129,"threads_started":1}
I20260620 21:56:14.997841 27980 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } }
I20260620 21:56:14.998423 27975 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:56:14.999042 27980 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3641c62ec0f45fd8973f0634546ba64. Latest consensus state: current_term: 1 leader_uuid: "d3641c62ec0f45fd8973f0634546ba64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } } }
I20260620 21:56:14.999150 27980 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:14.999475 27981 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3641c62ec0f45fd8973f0634546ba64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3641c62ec0f45fd8973f0634546ba64" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 36865 } } }
I20260620 21:56:14.999558 27981 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64 [sys.catalog]: This master's current role is: LEADER
I20260620 21:56:15.000708 27986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:56:15.003355 27986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:56:15.008821 27986 catalog_manager.cc:1347] Generated new cluster ID: 3f3169dbe27b470ab8c9bb49f5dee3bd
I20260620 21:56:15.008914 27986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:56:15.021394 27986 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:56:15.022367 27986 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:56:15.028824 27986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: Generated new TSK 0
I20260620 21:56:15.029502 27986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:56:15.062744 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:56:15.183295 28001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:15.183552 28001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:15.183593 28001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:15.187572 28001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:15.187628 28001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:15.187691 28001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:56:15.192826 28001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:15.193835 28001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:15.194798 28001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:15.204589 28006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.204661 28001 server_base.cc:1034] running on GCE node
W20260620 21:56:15.204586 28007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:15.204895 28009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.205238 28001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:15.205932 28001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:15.207154 28001 hybrid_clock.cc:648] HybridClock initialized: now 1781992575207120 us; error 52 us; skew 500 ppm
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:56:15.209990 28001 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:56:15.210819 28001 webserver.cc:460] Webserver started at http://127.17.67.193:43613/ using document root <none> and password file <none>
I20260620 21:56:15.211395 28001 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:15.211488 28001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:15.211714 28001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:15.213768 28001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "06ed4f9efe3140eb9ab2c24055cae665"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "883fdf66c8a3ac90c4648e5c215e2cef"
server_key_iv: "19437aa8a527eacd039b1083fc247de3"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.214316 28001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "06ed4f9efe3140eb9ab2c24055cae665"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "883fdf66c8a3ac90c4648e5c215e2cef"
server_key_iv: "19437aa8a527eacd039b1083fc247de3"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.217919 28001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:56:15.220268 28016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.221462 28001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:56:15.221629 28001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "06ed4f9efe3140eb9ab2c24055cae665"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "883fdf66c8a3ac90c4648e5c215e2cef"
server_key_iv: "19437aa8a527eacd039b1083fc247de3"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.221746 28001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:15.237243 28001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:15.240168 28001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:15.240376 28001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:15.240939 28001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:15.241839 28001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:15.241906 28001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.241969 28001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:15.242022 28001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.251905 28001 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:39897
I20260620 21:56:15.251927 28129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:39897 every 8 connection(s)
I20260620 21:56:15.252840 28001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:56:15.259418 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 28001
I20260620 21:56:15.259558 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:56:15.259856 17679 external_mini_cluster.cc:1428] Setting key a215f54ce28986baee4ea4760b7406c5
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:56:15.267270 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.254641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:44675 (local address 127.17.67.254:36865)
0620 21:56:15.254803 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.254808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.255641 (+ 833us) server_negotiation.cc:407] Connection header received
0620 21:56:15.256974 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.256977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.257021 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.257098 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:15.258354 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.258898 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.260362 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.260626 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.260711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.263550 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:15.263555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:15.263559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:56:15.263592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:15.265104 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.265709 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.265714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.265717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.265800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.266133 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.266136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.266138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.266359 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:15.266466 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.266960 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.267093 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:56:15.268097 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.255057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36865 (local address 127.17.67.193:44675)
0620 21:56:15.255490 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.255532 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.256753 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.257261 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.257269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.257709 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:15.258200 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.258214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.259017 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.259020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.260186 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.260197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.260773 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.260777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.260997 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.261675 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:15.261691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:15.263377 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:15.265232 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.265238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.265255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.265559 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.265895 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.265899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.265901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.266033 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.266463 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:15.266470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:15.266729 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.266907 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.267223 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260620 21:56:15.269310 28130 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36865
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:56:15.269593 28130 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:15.270071 28130 heartbeater.cc:511] Master 127.17.67.254:36865 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:56:15.271692 27939 ts_manager.cc:194] Registered new tserver with Master: 06ed4f9efe3140eb9ab2c24055cae665 (127.17.67.193:39897)
I20260620 21:56:15.272784 27939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:44675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:56:15.286046 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:56:15.393667 28135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:15.393925 28135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:15.393967 28135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:15.397940 28135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:15.397997 28135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:15.398054 28135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:56:15.404657 28135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:15.405824 28135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:15.406863 28135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:15.416435 28141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:15.416358 28143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.416407 28135 server_base.cc:1034] running on GCE node
W20260620 21:56:15.416309 28140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.416922 28135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:15.417654 28135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:15.418846 28135 hybrid_clock.cc:648] HybridClock initialized: now 1781992575418835 us; error 30 us; skew 500 ppm
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:56:15.421738 28135 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:56:15.422554 28135 webserver.cc:460] Webserver started at http://127.17.67.194:43549/ using document root <none> and password file <none>
I20260620 21:56:15.423113 28135 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:15.423205 28135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:15.423431 28135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:15.425549 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "14ab6ff93e24479184446345b23d7c2f"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "e92644be568866cd23f684f868d0a430"
server_key_iv: "0205193db17f40888acec796230d8ea0"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.426205 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "14ab6ff93e24479184446345b23d7c2f"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "e92644be568866cd23f684f868d0a430"
server_key_iv: "0205193db17f40888acec796230d8ea0"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.429876 28135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:56:15.432320 28150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.433642 28135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:56:15.433780 28135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "14ab6ff93e24479184446345b23d7c2f"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "e92644be568866cd23f684f868d0a430"
server_key_iv: "0205193db17f40888acec796230d8ea0"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.433888 28135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:15.479162 28135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:15.482646 28135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:15.482860 28135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:15.483472 28135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:15.484304 28135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:15.484373 28135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.484445 28135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:15.484488 28135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.492978 28135 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:33899
I20260620 21:56:15.492998 28263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:33899 every 8 connection(s)
I20260620 21:56:15.493871 28135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:56:15.502450 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 28135
I20260620 21:56:15.502573 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:56:15.502877 17679 external_mini_cluster.cc:1428] Setting key c30c6e947ca24ce709dcaed242fa8e1a
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:56:15.508673 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.495648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:56775 (local address 127.17.67.254:36865)
0620 21:56:15.495797 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.495801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.496824 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 21:56:15.498387 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.498390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.498435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.498524 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:15.499776 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.500361 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.501508 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.501811 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.501900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.504682 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:15.504688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:15.504692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:56:15.504718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:15.506411 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.507014 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.507016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.507018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.507087 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.507476 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.507482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.507486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.507725 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:15.507839 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.508284 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.508424 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:56:15.509284 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.496072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36865 (local address 127.17.67.194:56775)
0620 21:56:15.496629 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.496696 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.498184 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.498705 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.498711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.499069 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:15.499594 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.499614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.500469 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.500472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.501398 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.501407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.501935 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.501937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.502070 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.502780 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:15.502801 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:15.504523 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:15.506516 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.506522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.506539 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.506909 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.507177 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.507182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.507184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.507362 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.507832 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:15.507837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:15.508051 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.508203 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.508444 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260620 21:56:15.510478 28264 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36865
I20260620 21:56:15.510703 28264 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:15.511157 28264 heartbeater.cc:511] Master 127.17.67.254:36865 requested a full tablet report, sending...
I20260620 21:56:15.512125 27939 ts_manager.cc:194] Registered new tserver with Master: 14ab6ff93e24479184446345b23d7c2f (127.17.67.194:33899)
I20260620 21:56:15.512670 27939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:56775
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:56:15.529801 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:56:15.640257 28269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:56:15.640515 28269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:56:15.640556 28269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:56:15.644434 28269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:56:15.644491 28269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:56:15.644548 28269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:56:15.649909 28269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:56:15.651094 28269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:56:15.652150 28269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:56:15.661892 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.661967 28269 server_base.cc:1034] running on GCE node
W20260620 21:56:15.661917 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:56:15.661914 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:56:15.662582 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:56:15.663233 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:56:15.664434 28269 hybrid_clock.cc:648] HybridClock initialized: now 1781992575664396 us; error 50 us; skew 500 ppm
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:56:15.667272 28269 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:56:15.668174 28269 webserver.cc:460] Webserver started at http://127.17.67.195:41733/ using document root <none> and password file <none>
I20260620 21:56:15.668767 28269 fs_manager.cc:359] Metadata directory not provided
I20260620 21:56:15.668864 28269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:56:15.669096 28269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:56:15.671226 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "af81a2a9cef148d19655f38c8993f9f5"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "dae3a785da1afc32cd763ab3a7de6273"
server_key_iv: "96011ab1fcc8dd95f52c9183a67820c1"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.671798 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "af81a2a9cef148d19655f38c8993f9f5"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "dae3a785da1afc32cd763ab3a7de6273"
server_key_iv: "96011ab1fcc8dd95f52c9183a67820c1"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.675740 28269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:56:15.678224 28284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.679709 28269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:56:15.679857 28269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "af81a2a9cef148d19655f38c8993f9f5"
format_stamp: "Formatted at 2026-06-20 21:56:15 on dist-test-slave-cx8d"
server_key: "dae3a785da1afc32cd763ab3a7de6273"
server_key_iv: "96011ab1fcc8dd95f52c9183a67820c1"
server_key_version: "encryptionkey@0"
I20260620 21:56:15.679975 28269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:56:15.699002 28269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:56:15.702414 28269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:56:15.702622 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:56:15.703219 28269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:56:15.704049 28269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:56:15.704121 28269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.704195 28269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:56:15.704237 28269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:56:15.713290 28269 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:41497
I20260620 21:56:15.713332 28397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:41497 every 8 connection(s)
I20260620 21:56:15.714224 28269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:56:15.716022 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 28269
I20260620 21:56:15.716144 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:56:15.716401 17679 external_mini_cluster.cc:1428] Setting key f0c98daff030d618e75c10998df44859
Jun 20 21:56:15 dist-test-slave-cx8d krb5kdc[27894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992575, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:56:15.728349 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.716193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:59493 (local address 127.17.67.254:36865)
0620 21:56:15.716380 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.716385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.717279 (+ 894us) server_negotiation.cc:407] Connection header received
0620 21:56:15.718794 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.718797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.718842 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.718916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:15.720025 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.720542 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.721768 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.721978 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.722059 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.724596 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:15.724599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:15.724601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:56:15.724624 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:15.726427 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.726907 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.726909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.726910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.726984 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.727347 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.727350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.727351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.727535 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:15.727623 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.728067 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.728168 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 21:56:15.729374 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.716598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:36865 (local address 127.17.67.195:59493)
0620 21:56:15.717120 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.717172 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.718573 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.719050 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.719057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.719412 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:56:15.719900 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.719913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.720668 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.720672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.721654 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.721663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.722109 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.722112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.722287 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.722823 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:56:15.722839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:56:15.724430 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:56:15.726533 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.726538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.726550 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.726818 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.727068 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:56:15.727070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:56:15.727071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:56:15.727209 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:56:15.727635 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:56:15.727640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:56:15.727867 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.728027 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.728279 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":159,"threads_started":1}
I20260620 21:56:15.730571 28398 heartbeater.cc:347] Connected to a master server at 127.17.67.254:36865
I20260620 21:56:15.730815 28398 heartbeater.cc:464] Registering TS with master...
I20260620 21:56:15.731317 28398 heartbeater.cc:511] Master 127.17.67.254:36865 requested a full tablet report, sending...
I20260620 21:56:15.732384 27938 ts_manager.cc:194] Registered new tserver with Master: af81a2a9cef148d19655f38c8993f9f5 (127.17.67.195:41497)
I20260620 21:56:15.732942 27938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:59493
I20260620 21:56:15.742825 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:56:15.753997 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.745225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.17.67.254:36865)
0620 21:56:15.745378 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.745382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.745561 (+ 179us) server_negotiation.cc:407] Connection header received
0620 21:56:15.745762 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.745765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.745810 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.745887 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:56:15.746832 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.747387 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.748392 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.748622 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.748707 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.750587 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:56:15.750591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:56:15.750593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:56:15.750620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:56:15.752169 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.752642 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.752645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.752646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.752722 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:56:15.752984 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:56:15.752986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:56:15.752987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:56:15.753207 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:56:15.753345 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.753690 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.753837 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:56:15.757155 27938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:56:15.760113 27938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:56:15.774788 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.768764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.17.67.195:41497)
0620 21:56:15.769760 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.769774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.769792 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:56:15.769867 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.769872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.770080 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.770628 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:56:15.770796 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.771939 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.773418 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.774065 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.774210 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.774374 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.774529 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.774622 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":169,"threads_started":1}
I20260620 21:56:15.774820 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.768483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:41497 (local address 127.0.0.1:43882)
0620 21:56:15.769297 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.769316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.769452 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.770277 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.770283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.770316 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:56:15.770487 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.770495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.772111 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.772113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.773297 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.773306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.774202 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.774206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.774325 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.774368 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.774490 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.774618 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":43,"spinlock_wait_cycles":7296,"thread_start_us":112,"threads_started":1}
I20260620 21:56:15.775425 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.768546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:39897 (local address 127.0.0.1:49586)
0620 21:56:15.769263 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.769316 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.769452 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.770277 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.770283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.770318 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:56:15.770628 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.770634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.772072 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.772075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.773518 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.773528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.774764 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.774767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.774885 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.775146 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.775198 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.775275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":229,"spinlock_wait_cycles":14080,"thread_start_us":98,"threads_started":1}
I20260620 21:56:15.775573 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.770502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.17.67.194:33899)
0620 21:56:15.770922 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.770931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.770948 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:56:15.771008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.771012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.771157 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.771336 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:56:15.771541 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.772591 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.774141 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.774964 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.775079 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.775205 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.775341 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.775435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":139,"threads_started":1}
I20260620 21:56:15.775530 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.770081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33899 (local address 127.0.0.1:37670)
0620 21:56:15.770577 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:15.770591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:15.770690 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:15.771303 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:15.771306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:15.771324 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:56:15.771420 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.771427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.772746 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.772749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.774008 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:15.774018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.775102 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.775106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:15.775205 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.775223 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:56:15.775307 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:56:15.775368 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":111,"threads_started":1}
I20260620 21:56:15.776429 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:15.768822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.17.67.193:39897)
0620 21:56:15.769751 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:15.769767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:15.769792 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:56:15.769867 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:15.769872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:15.770080 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:15.770412 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:56:15.770748 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.771944 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.773693 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:15.774624 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:15.775942 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:15.776093 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:15.776148 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:56:15.776261 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":79,"threads_started":1}
I20260620 21:56:15.778553 28198 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3f41d9ba34427ab55538f966e40cd (DEFAULT_TABLE table=test-table [id=39a3bc6eaa5d4acf9bab621727a187e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:56:15.778877 28332 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3f41d9ba34427ab55538f966e40cd (DEFAULT_TABLE table=test-table [id=39a3bc6eaa5d4acf9bab621727a187e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:56:15.779740 28332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3f41d9ba34427ab55538f966e40cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:15.779740 28198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3f41d9ba34427ab55538f966e40cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:15.780167 28064 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3f41d9ba34427ab55538f966e40cd (DEFAULT_TABLE table=test-table [id=39a3bc6eaa5d4acf9bab621727a187e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:56:15.781042 28064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3f41d9ba34427ab55538f966e40cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:56:15.787226 28415 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Bootstrap starting.
I20260620 21:56:15.787693 28416 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Bootstrap starting.
I20260620 21:56:15.788306 28417 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Bootstrap starting.
I20260620 21:56:15.790042 28416 tablet_bootstrap.cc:654] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:15.790158 28417 tablet_bootstrap.cc:654] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:15.791115 28417 log.cc:824] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:15.791481 28416 log.cc:824] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:15.792387 28415 tablet_bootstrap.cc:654] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Neither blocks nor log segments found. Creating new log.
I20260620 21:56:15.792999 28417 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: No bootstrap required, opened a new log
I20260620 21:56:15.793126 28415 log.cc:824] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:56:15.793223 28417 ts_tablet_manager.cc:1397] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:56:15.795168 28416 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: No bootstrap required, opened a new log
I20260620 21:56:15.795162 28415 tablet_bootstrap.cc:492] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: No bootstrap required, opened a new log
I20260620 21:56:15.795375 28415 ts_tablet_manager.cc:1397] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 21:56:15.795375 28416 ts_tablet_manager.cc:1397] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 21:56:15.796908 28417 raft_consensus.cc:348] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.797231 28417 raft_consensus.cc:374] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:15.797317 28417 raft_consensus.cc:729] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ed4f9efe3140eb9ab2c24055cae665, State: Initialized, Role: FOLLOWER
I20260620 21:56:15.797835 28417 consensus_queue.cc:260] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.798707 28130 heartbeater.cc:503] Master 127.17.67.254:36865 was elected leader, sending a full tablet report...
I20260620 21:56:15.799005 28417 ts_tablet_manager.cc:1428] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:56:15.799672 28415 raft_consensus.cc:348] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.799674 28416 raft_consensus.cc:348] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.799961 28416 raft_consensus.cc:374] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:15.799961 28415 raft_consensus.cc:374] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:56:15.800027 28415 raft_consensus.cc:729] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ab6ff93e24479184446345b23d7c2f, State: Initialized, Role: FOLLOWER
I20260620 21:56:15.800069 28416 raft_consensus.cc:729] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af81a2a9cef148d19655f38c8993f9f5, State: Initialized, Role: FOLLOWER
I20260620 21:56:15.800518 28415 consensus_queue.cc:260] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.800518 28416 consensus_queue.cc:260] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:15.801411 28398 heartbeater.cc:503] Master 127.17.67.254:36865 was elected leader, sending a full tablet report...
I20260620 21:56:15.801626 28416 ts_tablet_manager.cc:1428] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 21:56:15.801622 28415 ts_tablet_manager.cc:1428] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 21:56:15.802170 28264 heartbeater.cc:503] Master 127.17.67.254:36865 was elected leader, sending a full tablet report...
W20260620 21:56:15.965240 28399 tablet.cc:2369] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:56:15.995128 28265 tablet.cc:2369] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:56:16.004130 28131 tablet.cc:2369] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:56:16.032262 28422 raft_consensus.cc:482] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:56:16.032466 28422 raft_consensus.cc:504] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:16.033776 28422 leader_election.cc:288] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af81a2a9cef148d19655f38c8993f9f5 (127.17.67.195:41497), 06ed4f9efe3140eb9ab2c24055cae665 (127.17.67.193:39897)
I20260620 21:56:16.039335 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.034157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:55677 (local address 127.17.67.193:39897)
0620 21:56:16.034308 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:16.034319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:16.034453 (+ 134us) server_negotiation.cc:407] Connection header received
0620 21:56:16.034527 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:16.034530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:16.034585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:16.034679 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:56:16.035080 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.035881 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.037839 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.038497 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.038669 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:16.038725 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:16.039113 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:56:16.039203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:56:16.039338 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.033979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:32861 (local address 127.17.67.195:41497)
0620 21:56:16.034167 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:16.034172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:16.034453 (+ 281us) server_negotiation.cc:407] Connection header received
0620 21:56:16.034527 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:16.034530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:16.034585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:16.034696 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:56:16.035080 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.035878 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.037839 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.038561 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.038677 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:16.038725 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:16.039113 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:56:16.039239 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 21:56:16.039443 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.034017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:39897 (local address 127.17.67.194:55677)
0620 21:56:16.034314 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:16.034329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:16.034441 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:16.034717 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:16.034719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:16.034746 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:56:16.034968 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:16.034974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.036007 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.036011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:16.037685 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:16.037696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.038670 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.038674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:16.038884 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:16.038971 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:56:16.039059 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 21:56:16.039204 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":367,"spinlock_wait_cycles":20608,"thread_start_us":97,"threads_started":1}
I20260620 21:56:16.039445 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.033914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:41497 (local address 127.17.67.194:32861)
0620 21:56:16.034287 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:16.034301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:56:16.034441 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:56:16.034716 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:56:16.034719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:56:16.034745 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:56:16.034957 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:16.034966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.036007 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.036011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:16.037688 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:56:16.037696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.038679 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.038682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:56:16.038904 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:16.038970 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 21:56:16.039059 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:56:16.039226 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":362,"spinlock_wait_cycles":20352,"thread_start_us":80,"threads_started":1}
I20260620 21:56:16.040087 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3f41d9ba34427ab55538f966e40cd" candidate_uuid: "14ab6ff93e24479184446345b23d7c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af81a2a9cef148d19655f38c8993f9f5" is_pre_election: true
I20260620 21:56:16.040346 28084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3f41d9ba34427ab55538f966e40cd" candidate_uuid: "14ab6ff93e24479184446345b23d7c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ed4f9efe3140eb9ab2c24055cae665" is_pre_election: true
I20260620 21:56:16.040431 28352 raft_consensus.cc:2454] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ab6ff93e24479184446345b23d7c2f in term 0.
I20260620 21:56:16.040609 28084 raft_consensus.cc:2454] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ab6ff93e24479184446345b23d7c2f in term 0.
I20260620 21:56:16.040906 28152 leader_election.cc:302] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ab6ff93e24479184446345b23d7c2f, af81a2a9cef148d19655f38c8993f9f5; no voters:
I20260620 21:56:16.041190 28422 raft_consensus.cc:2781] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:56:16.041306 28422 raft_consensus.cc:482] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:56:16.041370 28422 raft_consensus.cc:3037] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:16.042407 28422 raft_consensus.cc:504] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:16.042786 28422 leader_election.cc:288] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [CANDIDATE]: Term 1 election: Requested vote from peers af81a2a9cef148d19655f38c8993f9f5 (127.17.67.195:41497), 06ed4f9efe3140eb9ab2c24055cae665 (127.17.67.193:39897)
I20260620 21:56:16.043186 28084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3f41d9ba34427ab55538f966e40cd" candidate_uuid: "14ab6ff93e24479184446345b23d7c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06ed4f9efe3140eb9ab2c24055cae665"
I20260620 21:56:16.043314 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3f41d9ba34427ab55538f966e40cd" candidate_uuid: "14ab6ff93e24479184446345b23d7c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af81a2a9cef148d19655f38c8993f9f5"
I20260620 21:56:16.043315 28084 raft_consensus.cc:3037] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:16.043459 28352 raft_consensus.cc:3037] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:56:16.044574 28084 raft_consensus.cc:2454] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ab6ff93e24479184446345b23d7c2f in term 1.
I20260620 21:56:16.044606 28352 raft_consensus.cc:2454] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ab6ff93e24479184446345b23d7c2f in term 1.
I20260620 21:56:16.044871 28151 leader_election.cc:302] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06ed4f9efe3140eb9ab2c24055cae665, 14ab6ff93e24479184446345b23d7c2f; no voters:
I20260620 21:56:16.045073 28422 raft_consensus.cc:2781] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:56:16.045272 28422 raft_consensus.cc:686] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 1 LEADER]: Becoming Leader. State: Replica: 14ab6ff93e24479184446345b23d7c2f, State: Running, Role: LEADER
I20260620 21:56:16.045608 28422 consensus_queue.cc:237] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } }
I20260620 21:56:16.048836 27938 catalog_manager.cc:5495] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ab6ff93e24479184446345b23d7c2f (127.17.67.194). New cstate: current_term: 1 leader_uuid: "14ab6ff93e24479184446345b23d7c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ab6ff93e24479184446345b23d7c2f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33899 } health_report { overall_health: HEALTHY } } }
I20260620 21:56:16.067242 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.063723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.17.67.194:33899)
0620 21:56:16.063873 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:56:16.063878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:56:16.063892 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:56:16.063995 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:56:16.063997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:56:16.064048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:56:16.064161 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:56:16.064410 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.064906 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.065821 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:56:16.066031 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:56:16.066137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:56:16.066437 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:56:16.066884 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:56:16.067010 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:56:16.067057 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 21:56:16.067107 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:56:16.072933 28084 raft_consensus.cc:1261] T 06b3f41d9ba34427ab55538f966e40cd P 06ed4f9efe3140eb9ab2c24055cae665 [term 1 FOLLOWER]: Refusing update from remote peer 14ab6ff93e24479184446345b23d7c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:56:16.072932 28352 raft_consensus.cc:1261] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 1 FOLLOWER]: Refusing update from remote peer 14ab6ff93e24479184446345b23d7c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:56:16.073688 28426 consensus_queue.cc:1035] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af81a2a9cef148d19655f38c8993f9f5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:56:16.073848 28422 consensus_queue.cc:1035] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06ed4f9efe3140eb9ab2c24055cae665" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:56:16.082019 28430 mvcc.cc:205] Tried to move back new op lower bound from 7299041591587745792 to 7299041591485394944. Current Snapshot: MvccSnapshot[applied={T|T < 7299041591587745792}]
I20260620 21:56:16.083567 28431 mvcc.cc:205] Tried to move back new op lower bound from 7299041591587745792 to 7299041591485394944. Current Snapshot: MvccSnapshot[applied={T|T < 7299041591587745792}]
I20260620 21:56:16.091274 27939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:56:16.091470 27939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:56:16.093871 27939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3641c62ec0f45fd8973f0634546ba64: Sending DeleteTablet for 3 replicas of tablet 06b3f41d9ba34427ab55538f966e40cd
I20260620 21:56:16.094734 28198 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3f41d9ba34427ab55538f966e40cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:56:16 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:37670
I20260620 21:56:16.094739 28064 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3f41d9ba34427ab55538f966e40cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:56:16 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:49586
I20260620 21:56:16.095325 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 28001
I20260620 21:56:16.095391 28440 tablet_replica.cc:330] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: stopping tablet replica
I20260620 21:56:16.095703 28440 raft_consensus.cc:2229] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:56:16.096163 28440 raft_consensus.cc:2258] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:56:16.096062 28332 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3f41d9ba34427ab55538f966e40cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:56:16 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:43882
I20260620 21:56:16.097887 28440 ts_tablet_manager.cc:1905] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:56:16.098826 28441 tablet_replica.cc:330] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: stopping tablet replica
I20260620 21:56:16.099191 28441 raft_consensus.cc:2229] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:56:16.099642 28441 raft_consensus.cc:2258] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:56:16.101289 28441 ts_tablet_manager.cc:1905] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:56:16.101683 28440 ts_tablet_manager.cc:1918] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:56:16.101853 28440 log.cc:1196] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/06b3f41d9ba34427ab55538f966e40cd
I20260620 21:56:16.102269 28440 ts_tablet_manager.cc:1939] T 06b3f41d9ba34427ab55538f966e40cd P 14ab6ff93e24479184446345b23d7c2f: Deleting consensus metadata
I20260620 21:56:16.103534 27924 catalog_manager.cc:4841] TS 14ab6ff93e24479184446345b23d7c2f (127.17.67.194:33899): tablet 06b3f41d9ba34427ab55538f966e40cd (table test-table [id=39a3bc6eaa5d4acf9bab621727a187e4]) successfully deleted
I20260620 21:56:16.104076 28441 ts_tablet_manager.cc:1918] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:56:16.104239 28441 log.cc:1196] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/06b3f41d9ba34427ab55538f966e40cd
W20260620 21:56:16.104353 27923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:39897 (error 108)
I20260620 21:56:16.104593 28441 ts_tablet_manager.cc:1939] T 06b3f41d9ba34427ab55538f966e40cd P af81a2a9cef148d19655f38c8993f9f5: Deleting consensus metadata
I20260620 21:56:16.105365 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:56:16.105050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:39897 (local address 127.0.0.1:49588)
0620 21:56:16.105192 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 21:56:16.105266 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 21:56:16.105410 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 28135
I20260620 21:56:16.106294 27924 catalog_manager.cc:4841] TS af81a2a9cef148d19655f38c8993f9f5 (127.17.67.195:41497): tablet 06b3f41d9ba34427ab55538f966e40cd (table test-table [id=39a3bc6eaa5d4acf9bab621727a187e4]) successfully deleted
W20260620 21:56:16.106834 27923 catalog_manager.cc:4584] TS 06ed4f9efe3140eb9ab2c24055cae665 (127.17.67.193:39897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06b3f41d9ba34427ab55538f966e40cd: Network error: Client connection negotiation failed: client connection to 127.17.67.193:39897: connect: Connection refused (error 111)
I20260620 21:56:16.110956 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 28269
I20260620 21:56:16.116863 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 27907
2026-06-20T21:56:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2572 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:56:16 dist-test-slave-cx8d krb5kdc[28445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:56:16 dist-test-slave-cx8d krb5kdc[28445](info): set up 2 sockets
Jun 20 21:56:16 dist-test-slave-cx8d krb5kdc[28445](info): commencing operation
krb5kdc: starting...
W20260620 21:56:17.234491 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:56:17 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:56:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangeradmin_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangeradmin_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangerlookup_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangerlookup_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
I20260620 21:56:17.404158 17679 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiPMPYE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/postgres -l logfile start
2026-06-20 21:56:19.921 UTC [28475] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:56:19.921 UTC [28475] LOG: listening on IPv4 address "127.17.67.212", port 58277
2026-06-20 21:56:19.924 UTC [28475] LOG: listening on Unix socket "/tmp/.s.PGSQL.58277"
2026-06-20 21:56:19.947 UTC [28478] LOG: database system was shut down at 2026-06-20 21:56:18 UTC
2026-06-20 21:56:19.951 UTC [28475] LOG: database system is ready to accept connections
I20260620 21:56:20.986411 17679 mini_postgres.cc:96] Postgres bound to 58277
2026-06-20 21:56:20.992 UTC [28486] FATAL: database "slave" does not exist
127.17.67.212:58277 - accepting connections
I20260620 21:56:20.993603 17679 mini_ranger.cc:161] Starting Ranger...
I20260620 21:56:21.009231 17679 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:56:21.517611 17679 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:56:21.517712 17679 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin
2026-06-20 21:56:21,797 [I] DB FLAVOR :POSTGRES
2026-06-20 21:56:21,798 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:56:21,798 [I] Checking connection..
2026-06-20 21:56:21,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:56:22,070 [I] Checking connection passed.
2026-06-20 21:56:22,071 [I] --------- Verifying version history table ---------
2026-06-20 21:56:22,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:56:22,341 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:56:22,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:56:22,598 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:56:22,598 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:56:22,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:56:22.857 UTC [28550] WARNING: there is no transaction in progress
2026-06-20 21:56:22,864 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:56:22,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:56:23,117 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:56:23,117 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:56:23,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:56:23,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:56:23,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cx8d.c.gcp-upstream.internal','N') ;"
2026-06-20 21:56:23,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:56:24,053 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:56:24,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:56:24,303 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:56:24,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:56:24,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:56:24,781 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:56:24,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:56:25.733 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.750 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.766 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.780 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.794 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.812 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.865 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.873 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.890 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.901 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.911 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.925 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.934 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.940 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.947 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:25.954 UTC [28658] WARNING: there is no transaction in progress
2026-06-20 21:56:26,573 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:56:26,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:56:26,808 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:56:26,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:56:27,059 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:56:27,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:56:27,317 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:56:27,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:56:27,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:56:27,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cx8d.c.gcp-upstream.internal';"
2026-06-20 21:56:28,082 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:56:28,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:56:28,325 [I] DB_PATCHES have already been applied
I20260620 21:56:28.331969 17679 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiPMPYE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:56:28.332041 17679 mini_ranger.cc:193] Using host: 127.17.67.212
I20260620 21:56:28.335904 17679 mini_ranger.cc:227] Ranger admin URL: http://127.17.67.212:45371
Jun 20, 2026 9:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:56:29.478214 17679 mini_ranger.cc:160] Time spent starting Ranger: real 8.485s user 0.008s sys 0.003s
Jun 20, 2026 9:56:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:56:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 9:56:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:56:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:56:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:56:51.915493 17679 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangerkms_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/rangerkms_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.67.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.67.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:56:51 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:56:51.998399 17679 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:56:52.007896 17679 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:56:52.849128 17679 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:56:52.849249 17679 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms
I20260620 21:56:52.849354 17679 mini_ranger_kms.cc:227] Using postgres at 127.17.67.212:58277
2026-06-20 21:56:53,033 [I] DB FLAVOR :POSTGRES
2026-06-20 21:56:53,033 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:56:53,033 [I] Checking connection
2026-06-20 21:56:53,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:56:53,263 [I] connection success
2026-06-20 21:56:53,263 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:56:53,263 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:56:53,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:56:53,497 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:56:53,497 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:56:53,497 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:56:53,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:58277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:56:53,745 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:56:54.181768 17679 mini_ranger_kms.cc:310] Created kms service
I20260620 21:56:54.351225 17679 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:56:54.395064 17679 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:56:54 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:56:54.759845 17679 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:56:54.759990 17679 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:56:54.760010 17679 mini_ranger_kms.cc:240] Using host: 127.17.67.212
I20260620 21:56:54.763753 17679 mini_ranger_kms.cc:276] Ranger KMS PID: 28838
I20260620 21:56:54.763888 17679 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.67.212:48403
Jun 20, 2026 9:56:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:56:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:56:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:56:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:56:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:56:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 21:56:57.031775 17679 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.033s user 0.008s sys 0.159s
I20260620 21:56:57.031901 17679 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:56:57.031999 17679 mini_ranger_kms.cc:401] 127.17.67.212:48403/kms/v1/keys
Jun 20 21:56:57 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 21:56:57 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
Jun 20 21:56:57 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
Jun 20 21:57:00 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:57:00.258805 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45959
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:45959
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:48403/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:00.375416 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:00.375717 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:00.375758 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:00.379978 28889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:00.380053 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:00.380081 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:00.380098 28889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:00.380136 28889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:00.385963 28889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.67.212:48403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:45959
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:00.387104 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:00.388178 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:00.397358 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:00.397390 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:00.397749 28889 server_base.cc:1034] running on GCE node
W20260620 21:57:00.397367 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:00.398531 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:00.399888 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:00.401078 28889 hybrid_clock.cc:648] HybridClock initialized: now 1781992620401058 us; error 33 us; skew 500 ppm
Jun 20 21:57:00 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992620, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:00.404394 28889 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:00.405473 28889 webserver.cc:460] Webserver started at http://127.17.67.254:42779/ using document root <none> and password file <none>
I20260620 21:57:00.406082 28889 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:00.406196 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:00.406446 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:00 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992620, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:00.459172 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "0c3760b4c6cf4008af30652e136584fa"
format_stamp: "Formatted at 2026-06-20 21:57:00 on dist-test-slave-cx8d"
server_key: "7910838f048271110a69255c79e44e33"
server_key_iv: "9e7ec5eab3dd14c1234a762e972ec16f"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:00.459830 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "0c3760b4c6cf4008af30652e136584fa"
format_stamp: "Formatted at 2026-06-20 21:57:00 on dist-test-slave-cx8d"
server_key: "7910838f048271110a69255c79e44e33"
server_key_iv: "9e7ec5eab3dd14c1234a762e972ec16f"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:00.463922 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:57:00.478504 28904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:00.479916 28889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:57:00.480069 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "0c3760b4c6cf4008af30652e136584fa"
format_stamp: "Formatted at 2026-06-20 21:57:00 on dist-test-slave-cx8d"
server_key: "7910838f048271110a69255c79e44e33"
server_key_iv: "9e7ec5eab3dd14c1234a762e972ec16f"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:00.480209 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:00.508370 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:00.511626 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:00.511842 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:00.520228 28889 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:45959
I20260620 21:57:00.520274 28956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:45959 every 8 connection(s)
I20260620 21:57:00.521209 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:00.523949 28957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:00.526618 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 28889
I20260620 21:57:00.526762 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:57:00.527077 17679 external_mini_cluster.cc:1428] Setting key 533aa9a52ea85b3b20430f7653ce6419
I20260620 21:57:00.530174 28957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: Bootstrap starting.
I20260620 21:57:00.532681 28957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:00.533484 28957 log.cc:824] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:00.535741 28957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: No bootstrap required, opened a new log
Jun 20 21:57:00 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:00.540370 28957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } }
I20260620 21:57:00.540676 28957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:00.540792 28957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3760b4c6cf4008af30652e136584fa, State: Initialized, Role: FOLLOWER
I20260620 21:57:00.541354 28957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } }
I20260620 21:57:00.541556 28957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:57:00.541661 28957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:57:00.541774 28957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:00.543057 28957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } }
I20260620 21:57:00.543437 28957 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3760b4c6cf4008af30652e136584fa; no voters:
I20260620 21:57:00.543778 28957 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:57:00.543895 28962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:00.544183 28962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [term 1 LEADER]: Becoming Leader. State: Replica: 0c3760b4c6cf4008af30652e136584fa, State: Running, Role: LEADER
I20260620 21:57:00.544386 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:00.528591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.17.67.254:45959)
0620 21:57:00.529073 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:00.529104 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:00.529149 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:57:00.529974 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:00.529997 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:00.530421 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:00.530767 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:00.531645 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:00.533119 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:00.534362 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:00.535014 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:00.535185 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:00.538469 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:00.538477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:00.538498 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:00.538530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:00.540592 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:00.541252 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:00.541258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:00.541264 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:00.541398 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:00.541771 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:00.541775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:00.541778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:00.542313 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:00.542512 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:00.542832 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 21:57:00.543133 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":154,"threads_started":1}
I20260620 21:57:00.544560 28962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } }
I20260620 21:57:00.545233 28957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:00.545921 28962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3760b4c6cf4008af30652e136584fa. Latest consensus state: current_term: 1 leader_uuid: "0c3760b4c6cf4008af30652e136584fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } } }
I20260620 21:57:00.546041 28962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:00.545912 28963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3760b4c6cf4008af30652e136584fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3760b4c6cf4008af30652e136584fa" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 45959 } } }
I20260620 21:57:00.546453 28963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:00.547612 28966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:00.547848 28957 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:57:00.554141 28966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:57:00.561065 28966 catalog_manager.cc:1347] Generated new cluster ID: e7aec302b65a45869a96360d69dec1f8
I20260620 21:57:00.561189 28966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:00.579102 28966 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:57:00.580560 28966 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:00.593278 28966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: Generated new TSK 0
I20260620 21:57:00.594169 28966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:57:01.309443 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:01.432819 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:01.433125 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:01.433225 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:01.437655 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:01.437745 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:01.437849 29002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:57:01.444052 29002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.67.212:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:01.445231 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:01.446517 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:57:01.456451 29002 server_base.cc:1034] running on GCE node
W20260620 21:57:01.456437 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:01.456475 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:01.456709 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:01.457149 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:01.457892 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:01.459101 29002 hybrid_clock.cc:648] HybridClock initialized: now 1781992621459079 us; error 40 us; skew 500 ppm
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:01.462477 29002 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:57:01.463663 29002 webserver.cc:460] Webserver started at http://127.17.67.193:41451/ using document root <none> and password file <none>
I20260620 21:57:01.464450 29002 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:01.464558 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:01.464877 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:01.480224 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "637d45678a244a018faf3e01a34bccc7"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "55670da53fd7243566cf526e4ed27732"
server_key_iv: "a722be1d49e5826fb8e1b80c64941dda"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.480898 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "637d45678a244a018faf3e01a34bccc7"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "55670da53fd7243566cf526e4ed27732"
server_key_iv: "a722be1d49e5826fb8e1b80c64941dda"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.485677 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:57:01.503013 29018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.506326 29002 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 21:57:01.506587 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "637d45678a244a018faf3e01a34bccc7"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "55670da53fd7243566cf526e4ed27732"
server_key_iv: "a722be1d49e5826fb8e1b80c64941dda"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.506809 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:01.531126 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:01.534978 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:01.535235 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:01.536029 29002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:01.537137 29002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:01.537243 29002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.537365 29002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:01.537443 29002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.548702 29002 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:34937
I20260620 21:57:01.550861 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:57:01.553736 29132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:34937 every 8 connection(s)
I20260620 21:57:01.557449 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 29002
I20260620 21:57:01.557615 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:57:01.557958 17679 external_mini_cluster.cc:1428] Setting key 7f4d278f15fd0e1f4ce5784464f85d18
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:57:01.572420 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:01.557263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:39179 (local address 127.17.67.254:45959)
0620 21:57:01.559641 (+ 2378us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:01.559650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:01.559663 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:57:01.559702 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:01.559705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:01.559764 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:01.559850 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:01.561454 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.562325 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.564019 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.564431 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.564522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:01.567917 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:01.567921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:01.567926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:01.567961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:01.570045 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:01.570748 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:01.570752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:01.570754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:01.570845 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:01.571295 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:01.571301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:01.571305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:01.571510 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:01.571568 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:01.572173 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 21:57:01.572288 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2302,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:57:01.574007 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:01.556095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45959 (local address 127.17.67.193:39179)
0620 21:57:01.558169 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:01.558229 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:01.559235 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:01.560167 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:01.560178 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:01.560547 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:01.561269 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:01.561292 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.562469 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.562472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:01.563879 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:01.563890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.564751 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.564757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:01.564995 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:01.565803 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:01.565829 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:01.567675 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:01.570205 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:01.570212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:01.570231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:01.570614 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:01.570969 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:01.570974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:01.570976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:01.571172 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:01.571635 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:01.571640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:01.571889 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 21:57:01.572679 (+ 790us) client_negotiation.cc:240] Negotiation successful
0620 21:57:01.572979 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":149,"threads_started":1}
I20260620 21:57:01.576653 29133 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45959
I20260620 21:57:01.576895 29133 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:01.577490 29133 heartbeater.cc:511] Master 127.17.67.254:45959 requested a full tablet report, sending...
I20260620 21:57:01.579460 28921 ts_manager.cc:194] Registered new tserver with Master: 637d45678a244a018faf3e01a34bccc7 (127.17.67.193:34937)
I20260620 21:57:01.581187 28921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:39179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:57:01.598335 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:57:01.765693 29139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:01.766062 29139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:01.766135 29139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:01.772193 29139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:01.772284 29139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:01.772372 29139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:57:01.780289 29139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.67.212:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:01.781853 29139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:01.783347 29139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:01.796793 29150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:01.797695 29147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:01.796865 29148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:01.797170 29139 server_base.cc:1034] running on GCE node
I20260620 21:57:01.798497 29139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:01.799319 29139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:01.800712 29139 hybrid_clock.cc:648] HybridClock initialized: now 1781992621800638 us; error 92 us; skew 500 ppm
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:01.805009 29139 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:57:01.806285 29139 webserver.cc:460] Webserver started at http://127.17.67.194:44107/ using document root <none> and password file <none>
I20260620 21:57:01.807075 29139 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:01.807221 29139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:01.807485 29139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:01.827723 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "9c96492061f34382b958c5dedd24a08f"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "6e4931d8492408bc3457d3e10f9220c0"
server_key_iv: "26da2cd5cbc6abb05678b9e0059e1267"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.828549 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "9c96492061f34382b958c5dedd24a08f"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "6e4931d8492408bc3457d3e10f9220c0"
server_key_iv: "26da2cd5cbc6abb05678b9e0059e1267"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.833460 29139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 21:57:01.850193 29157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.851836 29139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:57:01.852005 29139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "9c96492061f34382b958c5dedd24a08f"
format_stamp: "Formatted at 2026-06-20 21:57:01 on dist-test-slave-cx8d"
server_key: "6e4931d8492408bc3457d3e10f9220c0"
server_key_iv: "26da2cd5cbc6abb05678b9e0059e1267"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:01.852174 29139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:01.866636 29139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:01.875640 29139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:01.876857 29139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:01.878770 29139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:01.879953 29139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:01.880028 29139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.880117 29139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:01.880155 29139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:01.892481 29139 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:39697
I20260620 21:57:01.893821 29139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:57:01.898561 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 29139
I20260620 21:57:01.898720 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:57:01.899065 17679 external_mini_cluster.cc:1428] Setting key 44631bf2630e22961e7df9cb25b80aea
I20260620 21:57:01.903585 29270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:39697 every 8 connection(s)
Jun 20 21:57:01 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:57:01.918926 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:01.895930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:53765 (local address 127.17.67.254:45959)
0620 21:57:01.896077 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:01.896082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:01.897182 (+ 1100us) server_negotiation.cc:407] Connection header received
0620 21:57:01.898533 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:01.898540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:01.898610 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:01.898682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:01.900528 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.901350 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.903415 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.903846 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.903950 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:01.914119 (+ 10169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:01.914125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:01.914128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:01.914179 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:01.916141 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:01.916879 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:01.916884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:01.916886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:01.916984 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:01.917470 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:01.917474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:01.917476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:01.917759 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:01.917868 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:01.918590 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 21:57:01.918740 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:57:01.920039 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:01.896453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45959 (local address 127.17.67.194:53765)
0620 21:57:01.896989 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:01.897050 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:01.898186 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:01.899197 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:01.899208 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:01.899602 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:01.900355 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:01.900377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.901531 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.901538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:01.903233 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:01.903251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:01.904210 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:01.904216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:01.904462 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:01.905288 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:01.905317 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:01.913357 (+ 8040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:01.916308 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:01.916316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:01.916337 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:01.916728 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:01.917094 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:01.917098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:01.917100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:01.917328 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:01.917876 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:01.917884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:01.918273 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 21:57:01.918502 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 21:57:01.918884 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":180,"threads_started":1}
I20260620 21:57:01.921386 29271 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45959
I20260620 21:57:01.921697 29271 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:01.922313 29271 heartbeater.cc:511] Master 127.17.67.254:45959 requested a full tablet report, sending...
I20260620 21:57:01.923535 28921 ts_manager.cc:194] Registered new tserver with Master: 9c96492061f34382b958c5dedd24a08f (127.17.67.194:39697)
I20260620 21:57:01.924362 28921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:53765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:57:01.941641 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:48403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:02.096040 29276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:02.096370 29276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:02.096473 29276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:02.102746 29276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:02.102840 29276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:02.102939 29276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:57:02.111758 29276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.67.212:48403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:45959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:02.113461 29276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:02.114970 29276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:02.129689 29281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:02.129704 29284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:02.130172 29282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:02.130563 29276 server_base.cc:1034] running on GCE node
I20260620 21:57:02.131093 29276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:02.132005 29276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:02.133208 29276 hybrid_clock.cc:648] HybridClock initialized: now 1781992622133188 us; error 37 us; skew 500 ppm
Jun 20 21:57:02 dist-test-slave-cx8d krb5kdc[28445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992622, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:02.140991 29276 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:57:02.142288 29276 webserver.cc:460] Webserver started at http://127.17.67.195:37995/ using document root <none> and password file <none>
I20260620 21:57:02.142994 29276 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:02.143076 29276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:02.143303 29276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:02 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992622, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:02.175218 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "9a803bca4e624115b92beafed37096a7"
format_stamp: "Formatted at 2026-06-20 21:57:02 on dist-test-slave-cx8d"
server_key: "8ee2ee72bad937eb15fe1755c2fc57ef"
server_key_iv: "5aba82307c147b21ceeeb4e4a3fe4a5b"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:02.177155 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "9a803bca4e624115b92beafed37096a7"
format_stamp: "Formatted at 2026-06-20 21:57:02 on dist-test-slave-cx8d"
server_key: "8ee2ee72bad937eb15fe1755c2fc57ef"
server_key_iv: "5aba82307c147b21ceeeb4e4a3fe4a5b"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:02.193940 29276 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.001s sys 0.003s
Jun 20 21:57:02 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:02.223552 29291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:02.225303 29276 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260620 21:57:02.225636 29276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "9a803bca4e624115b92beafed37096a7"
format_stamp: "Formatted at 2026-06-20 21:57:02 on dist-test-slave-cx8d"
server_key: "8ee2ee72bad937eb15fe1755c2fc57ef"
server_key_iv: "5aba82307c147b21ceeeb4e4a3fe4a5b"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:02.225813 29276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:02.245474 29276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:02.249213 29276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:02.249435 29276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:02.250257 29276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:02.251386 29276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:02.251458 29276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:02.251531 29276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:02.251562 29276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:02.264474 29276 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:41655
I20260620 21:57:02.265511 29276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:57:02.266440 29404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:41655 every 8 connection(s)
I20260620 21:57:02.267325 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 29276
I20260620 21:57:02.267457 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:57:02.267830 17679 external_mini_cluster.cc:1428] Setting key a4c8c45890f31dc13fd43d7fe8d67dc5
Jun 20 21:57:02 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992622, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:02.281347 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.269711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:53541 (local address 127.17.67.254:45959)
0620 21:57:02.269848 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.269854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.269880 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:57:02.270719 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.270723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.270777 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.270838 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:02.272441 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.273007 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.274444 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.274691 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.274813 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.277490 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:02.277495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:02.277498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:02.277528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:02.279162 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:02.279715 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:02.279721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:02.279723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:02.279814 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:02.280198 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:02.280204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:02.280206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:02.280398 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:02.280515 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.281032 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.281163 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:57:02.283032 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.268579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:45959 (local address 127.17.67.195:53541)
0620 21:57:02.269084 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.269143 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.270239 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.271252 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.271262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.271729 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:02.272243 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.272256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.273165 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.273169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.274241 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.274251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.274848 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.274853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.275095 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.275922 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:02.275940 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:02.277321 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:02.279306 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:02.279312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:02.279327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:02.279605 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:02.279921 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:02.279926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:02.279928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:02.280093 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:02.280512 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:02.280520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:02.280778 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.280975 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.281310 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":155,"threads_started":1}
I20260620 21:57:02.284418 29405 heartbeater.cc:347] Connected to a master server at 127.17.67.254:45959
I20260620 21:57:02.284689 29405 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:02.285310 29405 heartbeater.cc:511] Master 127.17.67.254:45959 requested a full tablet report, sending...
I20260620 21:57:02.286437 28921 ts_manager.cc:194] Registered new tserver with Master: 9a803bca4e624115b92beafed37096a7 (127.17.67.195:41655)
I20260620 21:57:02.287071 28921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:53541
I20260620 21:57:02.295419 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:57:02.306483 29136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.297757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.17.67.254:45959)
0620 21:57:02.297896 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.297901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.298196 (+ 295us) server_negotiation.cc:407] Connection header received
0620 21:57:02.298359 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.298363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.298411 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.298511 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:02.299273 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.299852 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.300766 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.301022 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.301149 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.302939 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:02.302943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:02.302946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:02.302973 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:02.304677 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:02.305183 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:02.305187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:02.305189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:02.305273 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:02.305560 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:02.305586 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:02.305588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:02.305805 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:02.305932 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.306206 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.306321 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:57:02.309861 28921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:57:02.314035 28921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:57:02.332207 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.323672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:34937 (local address 127.0.0.1:39608)
0620 21:57:02.326822 (+ 3150us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.326835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.326915 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.327497 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.327500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.327515 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:02.327594 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.327599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.328983 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.328986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.330437 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.330448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.331695 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.331698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.331805 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.331841 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.331945 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.332055 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3096,"mutex_wait_us":1196,"thread_start_us":94,"threads_started":1}
I20260620 21:57:02.332792 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.323930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.17.67.193:34937)
0620 21:57:02.326694 (+ 2764us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.326699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.326996 (+ 297us) server_negotiation.cc:407] Connection header received
0620 21:57:02.327069 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.327074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.327246 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.327373 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:02.327725 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.328854 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.330574 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.331389 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.331487 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.331624 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.332550 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.332645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2661,"thread_start_us":74,"threads_started":1}
I20260620 21:57:02.335352 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.323921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:41655 (local address 127.0.0.1:46278)
0620 21:57:02.325241 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.325290 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.325419 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.326252 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.326259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.326282 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:02.326456 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.326463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.329811 (+ 3348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.329856 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.331994 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.332009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.334065 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.334069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.334196 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.334226 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.335083 (+ 857us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.335182 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":3,"spinlock_wait_cycles":11520,"thread_start_us":93,"threads_started":1}
I20260620 21:57:02.335799 29049 tablet_service.cc:1467] Processing CreateTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c (DEFAULT_TABLE table=test-table [id=deb27395510440ba958eef514ea6c03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:02.336755 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.324618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.17.67.195:41655)
0620 21:57:02.325745 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.325759 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.325784 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:57:02.325857 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.325862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.326061 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.327896 (+ 1835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:02.328029 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.329678 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.332199 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.333914 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.336201 (+ 2287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.336440 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.336496 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.336596 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":125,"threads_started":1}
I20260620 21:57:02.336932 29049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39210ce5f39e4bca82e9b40b6a27df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:02.339821 29322 tablet_service.cc:1467] Processing CreateTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c (DEFAULT_TABLE table=test-table [id=deb27395510440ba958eef514ea6c03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:02.340819 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.323672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39697 (local address 127.0.0.1:48124)
0620 21:57:02.325240 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.325289 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.325419 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.326307 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.326310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.326328 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:02.326570 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.326574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.338215 (+ 11641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.338219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.339323 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.339335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.340431 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.340435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.340519 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.340554 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.340620 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.340685 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393,"mutex_wait_us":148,"spinlock_wait_cycles":4480,"thread_start_us":110,"threads_started":1}
I20260620 21:57:02.340842 29322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39210ce5f39e4bca82e9b40b6a27df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:02.341490 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.324263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.17.67.194:39697)
0620 21:57:02.325864 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.325875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.325897 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:57:02.325966 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.325971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.326157 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.326443 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:02.326687 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.330552 (+ 3865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.339517 (+ 8965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.340291 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.341078 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.341214 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.341264 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.341347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"thread_start_us":109,"threads_started":1}
I20260620 21:57:02.344106 29205 tablet_service.cc:1467] Processing CreateTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c (DEFAULT_TABLE table=test-table [id=deb27395510440ba958eef514ea6c03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:02.345126 29205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39210ce5f39e4bca82e9b40b6a27df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:02.348353 29423 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Bootstrap starting.
I20260620 21:57:02.351173 29423 tablet_bootstrap.cc:654] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:02.352110 29423 log.cc:824] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:02.353307 29424 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Bootstrap starting.
I20260620 21:57:02.355911 29424 tablet_bootstrap.cc:654] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:02.356709 29424 log.cc:824] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:02.358759 29423 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: No bootstrap required, opened a new log
I20260620 21:57:02.358959 29423 ts_tablet_manager.cc:1397] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260620 21:57:02.362999 29423 raft_consensus.cc:348] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.363256 29423 raft_consensus.cc:374] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:02.363680 29423 raft_consensus.cc:729] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a803bca4e624115b92beafed37096a7, State: Initialized, Role: FOLLOWER
I20260620 21:57:02.364014 29424 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: No bootstrap required, opened a new log
I20260620 21:57:02.364265 29424 ts_tablet_manager.cc:1397] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260620 21:57:02.364197 29423 consensus_queue.cc:260] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.365144 29423 ts_tablet_manager.cc:1428] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:57:02.365483 29405 heartbeater.cc:503] Master 127.17.67.254:45959 was elected leader, sending a full tablet report...
I20260620 21:57:02.368347 29424 raft_consensus.cc:348] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.369015 29427 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Bootstrap starting.
I20260620 21:57:02.369167 29424 raft_consensus.cc:374] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:02.369268 29424 raft_consensus.cc:729] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c96492061f34382b958c5dedd24a08f, State: Initialized, Role: FOLLOWER
I20260620 21:57:02.371901 29427 tablet_bootstrap.cc:654] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:02.372262 29424 consensus_queue.cc:260] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.373771 29424 ts_tablet_manager.cc:1428] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 21:57:02.373848 29271 heartbeater.cc:503] Master 127.17.67.254:45959 was elected leader, sending a full tablet report...
I20260620 21:57:02.375294 29427 log.cc:824] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:02.377549 29427 tablet_bootstrap.cc:492] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: No bootstrap required, opened a new log
I20260620 21:57:02.377789 29427 ts_tablet_manager.cc:1397] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 21:57:02.382261 29427 raft_consensus.cc:348] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.382521 29427 raft_consensus.cc:374] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:02.382567 29427 raft_consensus.cc:729] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637d45678a244a018faf3e01a34bccc7, State: Initialized, Role: FOLLOWER
I20260620 21:57:02.383024 29427 consensus_queue.cc:260] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.383806 29133 heartbeater.cc:503] Master 127.17.67.254:45959 was elected leader, sending a full tablet report...
I20260620 21:57:02.384255 29427 ts_tablet_manager.cc:1428] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260620 21:57:02.411347 29272 tablet.cc:2369] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:57:02 dist-test-slave-cx8d krb5kdc[28445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992621, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:02.502714 29431 raft_consensus.cc:482] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:57:02.502951 29431 raft_consensus.cc:504] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.504369 29431 leader_election.cc:288] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c96492061f34382b958c5dedd24a08f (127.17.67.194:39697), 9a803bca4e624115b92beafed37096a7 (127.17.67.195:41655)
I20260620 21:57:02.511078 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.504866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57929 (local address 127.17.67.195:41655)
0620 21:57:02.504988 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.504993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.505114 (+ 121us) server_negotiation.cc:407] Connection header received
0620 21:57:02.505357 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.505361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.505418 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.505506 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:02.505848 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.506579 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.509623 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.510190 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.510284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.510334 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.510857 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.510939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 21:57:02.511391 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.504579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:41655 (local address 127.17.67.193:57929)
0620 21:57:02.505023 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.505038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.505211 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.505539 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.505542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.505596 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:02.505744 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.505751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.506789 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.506794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.509443 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.509454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.510434 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.510438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.510701 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.510727 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.510823 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.510892 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":114,"threads_started":1}
I20260620 21:57:02.512042 29340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39210ce5f39e4bca82e9b40b6a27df2c" candidate_uuid: "637d45678a244a018faf3e01a34bccc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a803bca4e624115b92beafed37096a7" is_pre_election: true
I20260620 21:57:02.512399 29340 raft_consensus.cc:2454] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 637d45678a244a018faf3e01a34bccc7 in term 0.
I20260620 21:57:02.512938 29021 leader_election.cc:302] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637d45678a244a018faf3e01a34bccc7, 9a803bca4e624115b92beafed37096a7; no voters:
I20260620 21:57:02.513273 29431 raft_consensus.cc:2781] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:57:02.513363 29431 raft_consensus.cc:482] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:57:02.513401 29431 raft_consensus.cc:3037] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:02.515205 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.505001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59757 (local address 127.17.67.194:39697)
0620 21:57:02.505104 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.505109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.505541 (+ 432us) server_negotiation.cc:407] Connection header received
0620 21:57:02.505771 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.505774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.505829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.505877 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:02.507386 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.508140 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.513191 (+ 5051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.513757 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.513853 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.513917 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.515003 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.515086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:57:02.515166 29431 raft_consensus.cc:504] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.515517 29431 leader_election.cc:288] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [CANDIDATE]: Term 1 election: Requested vote from peers 9c96492061f34382b958c5dedd24a08f (127.17.67.194:39697), 9a803bca4e624115b92beafed37096a7 (127.17.67.195:41655)
I20260620 21:57:02.516018 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.504579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39697 (local address 127.17.67.193:59757)
0620 21:57:02.505454 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.505470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:02.505626 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:02.506190 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:02.506193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:02.506210 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:02.506619 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.506631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.508536 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.508539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.513054 (+ 4515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:02.513063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.514431 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.514434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:02.514850 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.514877 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:57:02.515413 (+ 536us) client_negotiation.cc:240] Negotiation successful
0620 21:57:02.515488 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":50,"thread_start_us":115,"threads_started":1}
I20260620 21:57:02.516057 29340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39210ce5f39e4bca82e9b40b6a27df2c" candidate_uuid: "637d45678a244a018faf3e01a34bccc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a803bca4e624115b92beafed37096a7"
I20260620 21:57:02.516191 29340 raft_consensus.cc:3037] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 0 FOLLOWER]: Advancing to term 1
W20260620 21:57:02.517302 29406 tablet.cc:2369] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:02.517390 29340 raft_consensus.cc:2454] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637d45678a244a018faf3e01a34bccc7 in term 1.
I20260620 21:57:02.518240 29225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39210ce5f39e4bca82e9b40b6a27df2c" candidate_uuid: "637d45678a244a018faf3e01a34bccc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c96492061f34382b958c5dedd24a08f" is_pre_election: true
I20260620 21:57:02.518534 29225 raft_consensus.cc:2454] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 637d45678a244a018faf3e01a34bccc7 in term 0.
I20260620 21:57:02.518872 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39210ce5f39e4bca82e9b40b6a27df2c" candidate_uuid: "637d45678a244a018faf3e01a34bccc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c96492061f34382b958c5dedd24a08f"
I20260620 21:57:02.518985 29216 raft_consensus.cc:3037] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:02.520246 29216 raft_consensus.cc:2454] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637d45678a244a018faf3e01a34bccc7 in term 1.
I20260620 21:57:02.520910 29023 leader_election.cc:302] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637d45678a244a018faf3e01a34bccc7, 9c96492061f34382b958c5dedd24a08f; no voters:
I20260620 21:57:02.521314 29431 raft_consensus.cc:2781] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:02.521513 29431 raft_consensus.cc:686] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [term 1 LEADER]: Becoming Leader. State: Replica: 637d45678a244a018faf3e01a34bccc7, State: Running, Role: LEADER
I20260620 21:57:02.521864 29431 consensus_queue.cc:237] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } }
I20260620 21:57:02.525091 28921 catalog_manager.cc:5495] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 637d45678a244a018faf3e01a34bccc7 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "637d45678a244a018faf3e01a34bccc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637d45678a244a018faf3e01a34bccc7" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 } health_report { overall_health: UNKNOWN } } }
W20260620 21:57:02.558609 29134 tablet.cc:2369] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:02.575809 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.570858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.17.67.193:34937)
0620 21:57:02.570980 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:02.570984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:02.571000 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:57:02.571083 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:02.571086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:02.571140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:02.571275 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:57:02.571541 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.572327 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.573620 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:02.573964 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:02.574060 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:02.574486 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:57:02.575041 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:57:02.575171 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:02.575598 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 21:57:02.575661 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:57:02.583380 29340 raft_consensus.cc:1261] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 1 FOLLOWER]: Refusing update from remote peer 637d45678a244a018faf3e01a34bccc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:02.584319 29216 raft_consensus.cc:1261] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 1 FOLLOWER]: Refusing update from remote peer 637d45678a244a018faf3e01a34bccc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:02.584728 29435 consensus_queue.cc:1035] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a803bca4e624115b92beafed37096a7" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:02.584995 29431 consensus_queue.cc:1035] T 39210ce5f39e4bca82e9b40b6a27df2c P 637d45678a244a018faf3e01a34bccc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c96492061f34382b958c5dedd24a08f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:02.604818 29439 mvcc.cc:205] Tried to move back new op lower bound from 7299041782092836864 to 7299041781852475392. Current Snapshot: MvccSnapshot[applied={T|T < 7299041782092836864}]
I20260620 21:57:02.606770 29440 mvcc.cc:205] Tried to move back new op lower bound from 7299041782092836864 to 7299041781852475392. Current Snapshot: MvccSnapshot[applied={T|T < 7299041782092836864}]
I20260620 21:57:02.614609 29442 mvcc.cc:205] Tried to move back new op lower bound from 7299041782092836864 to 7299041781852475392. Current Snapshot: MvccSnapshot[applied={T|T < 7299041782092836864}]
I20260620 21:57:02.625629 28921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:57:02.625820 28921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:57:02.628875 28921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c3760b4c6cf4008af30652e136584fa: Sending DeleteTablet for 3 replicas of tablet 39210ce5f39e4bca82e9b40b6a27df2c
I20260620 21:57:02.630116 29049 tablet_service.cc:1514] Processing DeleteTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:02 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:39608
I20260620 21:57:02.630622 29205 tablet_service.cc:1514] Processing DeleteTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:02 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:48124
I20260620 21:57:02.631846 29322 tablet_service.cc:1514] Processing DeleteTablet for tablet 39210ce5f39e4bca82e9b40b6a27df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:02 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:46278
I20260620 21:57:02.632645 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 29002
I20260620 21:57:02.635154 29450 tablet_replica.cc:330] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7: stopping tablet replica
I20260620 21:57:02.635517 29450 raft_consensus.cc:2229] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:57:02.635870 29450 raft_consensus.cc:2258] T 39210ce5f39e4bca82e9b40b6a27df2c P 9a803bca4e624115b92beafed37096a7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:57:02.641700 28905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:34937 (error 108)
I20260620 21:57:02.642092 29449 tablet_replica.cc:330] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f: stopping tablet replica
I20260620 21:57:02.642526 29449 raft_consensus.cc:2229] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:57:02.642904 29449 raft_consensus.cc:2258] T 39210ce5f39e4bca82e9b40b6a27df2c P 9c96492061f34382b958c5dedd24a08f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:57:02.643440 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.643088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:34937 (local address 127.0.0.1:39632)
0620 21:57:02.643284 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.643351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 21:57:02.644062 28905 catalog_manager.cc:4584] TS 637d45678a244a018faf3e01a34bccc7 (127.17.67.193:34937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39210ce5f39e4bca82e9b40b6a27df2c: Network error: Client connection negotiation failed: client connection to 127.17.67.193:34937: connect: Connection refused (error 111)
I20260620 21:57:02.645767 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 29139
I20260620 21:57:02.654922 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 29276
I20260620 21:57:02.656133 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.655829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39697 (local address 127.0.0.1:48140)
0620 21:57:02.655999 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.656048 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.194:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 21:57:02.662053 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:02.661808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:41655 (local address 127.0.0.1:46292)
0620 21:57:02.661916 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:02.661968 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.195:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 21:57:02.662317 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 28889
I20260620 21:57:02.670266 17679 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:57:03.015285 17679 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:57:03.015410 17679 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:57:03.360344 17679 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:57:03.360 UTC [28475] LOG: received smart shutdown request
2026-06-20 21:57:03.364 UTC [28475] LOG: background worker "logical replication launcher" (PID 28484) exited with exit code 1
2026-06-20 21:57:03.364 UTC [28479] LOG: shutting down
2026-06-20 21:57:03.526 UTC [28475] LOG: database system is shut down
2026-06-20T21:57:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47526 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:57:03 dist-test-slave-cx8d krb5kdc[29460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:57:03 dist-test-slave-cx8d krb5kdc[29460](info): set up 2 sockets
Jun 20 21:57:03 dist-test-slave-cx8d krb5kdc[29460](info): commencing operation
krb5kdc: starting...
W20260620 21:57:04.705650 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:57:04 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:57:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangeradmin_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangeradmin_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangerlookup_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangerlookup_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
I20260620 21:57:04.888406 17679 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiPMPYE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/postgres -l logfile start
2026-06-20 21:57:07.364 UTC [29490] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:57:07.364 UTC [29490] LOG: listening on IPv4 address "127.17.67.212", port 47633
2026-06-20 21:57:07.368 UTC [29490] LOG: listening on Unix socket "/tmp/.s.PGSQL.47633"
2026-06-20 21:57:07.402 UTC [29493] LOG: database system was shut down at 2026-06-20 21:57:05 UTC
2026-06-20 21:57:07.406 UTC [29490] LOG: database system is ready to accept connections
I20260620 21:57:08.357532 17679 mini_postgres.cc:96] Postgres bound to 47633
2026-06-20 21:57:08.364 UTC [29501] FATAL: database "slave" does not exist
127.17.67.212:47633 - accepting connections
I20260620 21:57:08.364677 17679 mini_ranger.cc:161] Starting Ranger...
I20260620 21:57:08.379606 17679 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:57:08.832533 17679 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:57:08.832625 17679 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin
2026-06-20 21:57:09,104 [I] DB FLAVOR :POSTGRES
2026-06-20 21:57:09,105 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:57:09,105 [I] Checking connection..
2026-06-20 21:57:09,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:57:09,347 [I] Checking connection passed.
2026-06-20 21:57:09,348 [I] --------- Verifying version history table ---------
2026-06-20 21:57:09,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:57:09,584 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:57:09,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:57:09,852 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:57:09,852 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:57:09,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:57:10.117 UTC [29565] WARNING: there is no transaction in progress
2026-06-20 21:57:10,124 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:57:10,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:57:10,388 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:57:10,388 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:57:10,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:57:10,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:57:10,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cx8d.c.gcp-upstream.internal','N') ;"
2026-06-20 21:57:11,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:57:11,464 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:57:11,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:57:11,717 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:57:11,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:57:11,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:57:12,213 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:57:12,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:57:13.208 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.226 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.244 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.260 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.275 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.296 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.354 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.362 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.376 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.387 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.397 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.411 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.421 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.428 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.435 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:13.442 UTC [29673] WARNING: there is no transaction in progress
2026-06-20 21:57:14,153 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:57:14,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:57:14,396 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:57:14,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:57:14,634 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:57:14,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:57:14,920 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:57:14,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:57:15,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:57:15,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cx8d.c.gcp-upstream.internal';"
2026-06-20 21:57:15,676 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:57:15,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:57:15,929 [I] DB_PATCHES have already been applied
I20260620 21:57:15.935113 17679 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiPMPYE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:57:15.935264 17679 mini_ranger.cc:193] Using host: 127.17.67.212
I20260620 21:57:15.939298 17679 mini_ranger.cc:227] Ranger admin URL: http://127.17.67.212:37375
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:57:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:57:17.109308 17679 mini_ranger.cc:160] Time spent starting Ranger: real 8.745s user 0.004s sys 0.006s
Jun 20, 2026 9:57:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:57:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 9:57:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:57:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:57:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:57:38.814078 17679 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.67.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.67.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangerkms_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.67.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/rangerkms_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.67.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.67.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.67.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/HTTP_127.17.67.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:57:38 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992658, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:57:38.898918 17679 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:57:38.909175 17679 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:57:39.813194 17679 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:57:39.813292 17679 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms
I20260620 21:57:39.813313 17679 mini_ranger_kms.cc:227] Using postgres at 127.17.67.212:47633
2026-06-20 21:57:40,002 [I] DB FLAVOR :POSTGRES
2026-06-20 21:57:40,002 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:57:40,002 [I] Checking connection
2026-06-20 21:57:40,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:57:40,263 [I] connection success
2026-06-20 21:57:40,264 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:57:40,264 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:57:40,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:57:40,518 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:57:40,519 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:57:40,519 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:57:40,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.67.212:47633/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:57:40,810 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:57:41.295068 17679 mini_ranger_kms.cc:310] Created kms service
I20260620 21:57:41.519637 17679 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:57:41.565892 17679 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:57:41 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992658, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:41.988521 17679 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:57:41.988795 17679 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiPMPYE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiPMPYE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:57:41.988823 17679 mini_ranger_kms.cc:240] Using host: 127.17.67.212
I20260620 21:57:41.992599 17679 mini_ranger_kms.cc:276] Ranger KMS PID: 29853
I20260620 21:57:41.992702 17679 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.67.212:42453
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:57:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 21:57:44.335445 17679 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.437s user 0.006s sys 0.167s
I20260620 21:57:44.335579 17679 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:57:44.335682 17679 mini_ranger_kms.cc:401] 127.17.67.212:42453/kms/v1/keys
Jun 20 21:57:44 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 21:57:44 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
Jun 20 21:57:45 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992664, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.67.212@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
Jun 20 21:57:47 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:57:47.785202 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41873
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41873
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:47.903098 29904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:47.903407 29904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:47.903452 29904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:47.907680 29904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:47.907752 29904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:47.907773 29904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:47.907790 29904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:47.907805 29904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:47.913455 29904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.67.212:42453/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41873
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:47.914539 29904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:47.915555 29904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:47.924525 29910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:47.924492 29912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:47.924588 29904 server_base.cc:1034] running on GCE node
W20260620 21:57:47.924537 29909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:47.925361 29904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:47.926605 29904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:47.927847 29904 hybrid_clock.cc:648] HybridClock initialized: now 1781992667927788 us; error 48 us; skew 500 ppm
Jun 20 21:57:47 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992667, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:47.930934 29904 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:47.931879 29904 webserver.cc:460] Webserver started at http://127.17.67.254:40457/ using document root <none> and password file <none>
I20260620 21:57:47.932431 29904 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:47.932550 29904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:47.932782 29904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:47 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992667, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:47.980665 29904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "84c1327fb3344077bea2a84f55f97904"
format_stamp: "Formatted at 2026-06-20 21:57:47 on dist-test-slave-cx8d"
server_key: "9bb754a03b9b07eafb1c9ed8415240da"
server_key_iv: "a91d140a59194de1d105123ce98f9756"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:47.981364 29904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "84c1327fb3344077bea2a84f55f97904"
format_stamp: "Formatted at 2026-06-20 21:57:47 on dist-test-slave-cx8d"
server_key: "9bb754a03b9b07eafb1c9ed8415240da"
server_key_iv: "a91d140a59194de1d105123ce98f9756"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:47.986001 29904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:57:48.000886 29919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:48.002415 29904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:57:48.002573 29904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "84c1327fb3344077bea2a84f55f97904"
format_stamp: "Formatted at 2026-06-20 21:57:47 on dist-test-slave-cx8d"
server_key: "9bb754a03b9b07eafb1c9ed8415240da"
server_key_iv: "a91d140a59194de1d105123ce98f9756"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:48.002734 29904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:48.041642 29904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:48.044754 29904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:48.044952 29904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:48.052765 29904 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41873
I20260620 21:57:48.052769 29971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41873 every 8 connection(s)
I20260620 21:57:48.053844 29904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:48.056632 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:48.060669 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 29904
I20260620 21:57:48.060804 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:57:48.061118 17679 external_mini_cluster.cc:1428] Setting key b19d7e8a11b12dc0d136b4f26b786af0
I20260620 21:57:48.062760 29972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Bootstrap starting.
I20260620 21:57:48.065527 29972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:48.066483 29972 log.cc:824] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:48.069708 29972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: No bootstrap required, opened a new log
Jun 20 21:57:48 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:48.074147 29972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:48.074415 29972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:48.074492 29972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c1327fb3344077bea2a84f55f97904, State: Initialized, Role: FOLLOWER
I20260620 21:57:48.074990 29972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:48.075145 29972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:57:48.075233 29972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:57:48.075341 29972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:48.076419 29972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:48.076812 29972 leader_election.cc:302] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c1327fb3344077bea2a84f55f97904; no voters:
I20260620 21:57:48.077188 29972 leader_election.cc:288] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:57:48.077250 29977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:48.077494 29977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 LEADER]: Becoming Leader. State: Replica: 84c1327fb3344077bea2a84f55f97904, State: Running, Role: LEADER
I20260620 21:57:48.077878 29977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:48.078141 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:48.062690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.17.67.254:41873)
0620 21:57:48.063153 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:48.063173 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:48.063216 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:57:48.064101 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:48.064128 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:48.064581 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:48.064948 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:48.065834 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:48.067190 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:48.068074 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:48.068775 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:48.068955 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:48.072169 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:48.072176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:48.072195 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:48.072229 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:48.074351 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:48.074910 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:48.074916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:48.074923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:48.075051 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:48.075373 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:48.075376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:48.075378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:48.075885 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:48.076104 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:48.076401 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 21:57:48.076865 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260620 21:57:48.078621 29972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:48.079231 29977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c1327fb3344077bea2a84f55f97904. Latest consensus state: current_term: 1 leader_uuid: "84c1327fb3344077bea2a84f55f97904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } } }
I20260620 21:57:48.079424 29977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:48.079663 29977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c1327fb3344077bea2a84f55f97904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } } }
I20260620 21:57:48.079802 29977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:48.080564 29981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:48.084077 29981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:57:48.084573 29972 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:57:48.090500 29981 catalog_manager.cc:1347] Generated new cluster ID: 7bf8e6ac41af4ada883aece838641f98
I20260620 21:57:48.090574 29981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:48.104840 29981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:57:48.106382 29981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:48.115854 29981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Generated new TSK 0
I20260620 21:57:48.116842 29981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:57:48.812824 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:48.933940 30017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:48.934191 30017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:48.934235 30017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:48.938382 30017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:48.938445 30017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:48.938503 30017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:57:48.944129 30017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:48.945267 30017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:48.946550 30017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:48.956157 30023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:48.956643 30022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:48.956825 30025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:48.957480 30017 server_base.cc:1034] running on GCE node
I20260620 21:57:48.957934 30017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:48.958642 30017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:48.959832 30017 hybrid_clock.cc:648] HybridClock initialized: now 1781992668959801 us; error 48 us; skew 500 ppm
Jun 20 21:57:48 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992668, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:48.963140 30017 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:57:48.964345 30017 webserver.cc:460] Webserver started at http://127.17.67.193:33089/ using document root <none> and password file <none>
I20260620 21:57:48.965076 30017 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:48.965181 30017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:48.965425 30017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:48 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992668, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:48.982178 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "a31c092f073947478797fc24f5fe81ab"
format_stamp: "Formatted at 2026-06-20 21:57:48 on dist-test-slave-cx8d"
server_key: "378beb3978e265492aa4f71b8043e9cc"
server_key_iv: "f69be5e372763ae088677f9b31123421"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:48.982950 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "a31c092f073947478797fc24f5fe81ab"
format_stamp: "Formatted at 2026-06-20 21:57:48 on dist-test-slave-cx8d"
server_key: "378beb3978e265492aa4f71b8043e9cc"
server_key_iv: "f69be5e372763ae088677f9b31123421"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:48.987972 30017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 21:57:49.014107 30034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.015739 30017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:57:49.015893 30017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "a31c092f073947478797fc24f5fe81ab"
format_stamp: "Formatted at 2026-06-20 21:57:48 on dist-test-slave-cx8d"
server_key: "378beb3978e265492aa4f71b8043e9cc"
server_key_iv: "f69be5e372763ae088677f9b31123421"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.016045 30017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:49.046326 30017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:49.049544 30017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:49.049803 30017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:49.050467 30017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:49.051373 30017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:49.051441 30017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.051555 30017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:49.051609 30017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.061508 30017 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:34315
I20260620 21:57:49.062757 30017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:57:49.072005 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30017
I20260620 21:57:49.072140 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:57:49.072485 17679 external_mini_cluster.cc:1428] Setting key 1da1c11352c84f63008edd31aa69c3e6
I20260620 21:57:49.064951 30147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:34315 every 8 connection(s)
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992668, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:49.084331 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.067460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.193:39163)
0620 21:57:49.069085 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:49.069147 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:49.070208 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:49.071206 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:49.071215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:49.071551 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:49.072234 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.072255 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.073323 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.073326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.074900 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.074911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.075679 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.075683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.075878 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.076560 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:49.076581 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:49.078445 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:49.080879 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.080886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.080903 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.081273 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.081634 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.081638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.081640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.081857 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.082390 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:49.082398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:49.082719 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 21:57:49.082916 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 21:57:49.083274 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":139,"threads_started":1}
I20260620 21:57:49.085100 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.066941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:39163 (local address 127.17.67.254:41873)
0620 21:57:49.070677 (+ 3736us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:49.070686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:49.070703 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:57:49.070748 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:49.070751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:49.070805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:49.070893 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:49.072414 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.073185 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.075041 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.075410 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.075511 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.078634 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:49.078638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:49.078641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:49.078668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:49.080717 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.081398 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.081402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.081404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.081498 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.082000 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.082004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.082006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.082246 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:49.082337 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:49.084775 (+ 2438us) server_negotiation.cc:299] Negotiation successful
0620 21:57:49.084915 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3649,"thread_start_us":115,"threads_started":1}
I20260620 21:57:49.086243 30148 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:49.086503 30148 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:49.087086 30148 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:49.088867 29936 ts_manager.cc:194] Registered new tserver with Master: a31c092f073947478797fc24f5fe81ab (127.17.67.193:34315)
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:57:49.090484 29936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:39163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:57:49.110853 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:57:49.272505 30156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:49.272895 30156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:49.272969 30156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:49.279241 30156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:49.279327 30156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:49.279414 30156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:57:49.287417 30156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:49.288879 30156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:49.290388 30156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:49.302058 30165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.302204 30163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.302081 30162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:49.302820 30156 server_base.cc:1034] running on GCE node
I20260620 21:57:49.303316 30156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:49.304095 30156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:49.305279 30156 hybrid_clock.cc:648] HybridClock initialized: now 1781992669305253 us; error 39 us; skew 500 ppm
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:49.308854 30156 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:57:49.309970 30156 webserver.cc:460] Webserver started at http://127.17.67.194:44939/ using document root <none> and password file <none>
I20260620 21:57:49.310669 30156 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:49.310755 30156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:49.311204 30156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:49.332744 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "5af62a614d4b4e878e5004099882b106"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "ac24ff425d3ff55e7c2ea736167df294"
server_key_iv: "dfd1af2c0ea8187ffb0b3c39520e0601"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.333504 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "5af62a614d4b4e878e5004099882b106"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "ac24ff425d3ff55e7c2ea736167df294"
server_key_iv: "dfd1af2c0ea8187ffb0b3c39520e0601"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.338958 30156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 21:57:49.354660 30172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.356411 30156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:57:49.356537 30156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "5af62a614d4b4e878e5004099882b106"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "ac24ff425d3ff55e7c2ea736167df294"
server_key_iv: "dfd1af2c0ea8187ffb0b3c39520e0601"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.356666 30156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:49.374843 30156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:49.378674 30156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:49.378865 30156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:49.379550 30156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:49.380637 30156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:49.380707 30156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.380762 30156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:49.380795 30156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.393250 30156 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:46555
I20260620 21:57:49.394552 30156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:57:49.397863 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30156
I20260620 21:57:49.398002 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:57:49.398337 17679 external_mini_cluster.cc:1428] Setting key 860ed5687715df7456048d1c3c57d8be
I20260620 21:57:49.402511 30285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:46555 every 8 connection(s)
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:57:49.439816 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.413167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:53519 (local address 127.17.67.254:41873)
0620 21:57:49.413291 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:49.413296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:49.413922 (+ 626us) server_negotiation.cc:407] Connection header received
0620 21:57:49.417324 (+ 3402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:49.417329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:49.417394 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:49.417502 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:49.419234 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.420080 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.421918 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.422310 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.422396 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.435005 (+ 12609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:49.435010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:49.435013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:49.435062 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:49.436959 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.437718 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.437725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.437726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.437813 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.438283 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.438286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.438288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.438526 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:49.438609 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:49.439465 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 21:57:49.439627 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:57:49.440317 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:57:49.441686 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.412023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.194:53519)
0620 21:57:49.413739 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:49.413802 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:49.417038 (+ 3236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:49.417870 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:49.417880 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:49.418276 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:49.419034 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.419058 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.420288 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.420291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.421772 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.421783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.422614 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.422618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.422822 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.425468 (+ 2646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:49.425490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:49.434730 (+ 9240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:49.437151 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.437159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.437179 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.437590 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.437927 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.437930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.437932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.438163 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.438736 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:49.438744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:49.439160 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 21:57:49.440068 (+ 908us) client_negotiation.cc:240] Negotiation successful
0620 21:57:49.440460 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"thread_start_us":155,"threads_started":1}
I20260620 21:57:49.443055 30286 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:49.443310 30286 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:49.443884 30286 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:49.445070 29936 ts_manager.cc:194] Registered new tserver with Master: 5af62a614d4b4e878e5004099882b106 (127.17.67.194:46555)
I20260620 21:57:49.445830 29936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:53519
W20260620 21:57:49.587220 30291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:49.587512 30291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:49.587631 30291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:49.592010 30291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:49.592108 30291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:49.592217 30291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:57:49.598873 30291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:49.599990 30291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:49.601023 30291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:49.612212 30299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.612785 30297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.612877 30296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:49.613796 30291 server_base.cc:1034] running on GCE node
I20260620 21:57:49.614261 30291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:49.615008 30291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:49.616189 30291 hybrid_clock.cc:648] HybridClock initialized: now 1781992669616166 us; error 40 us; skew 500 ppm
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:49.620218 30291 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:57:49.621490 30291 webserver.cc:460] Webserver started at http://127.17.67.195:39897/ using document root <none> and password file <none>
I20260620 21:57:49.622272 30291 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:49.622388 30291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:49.622651 30291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:49.653431 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "cebc303216474e8285712f418c301ae5"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "06096db0b0bd36f8bb7b1fea96e5ef19"
server_key_iv: "91d2148032a3274632a07aace69c8028"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.654297 30291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "cebc303216474e8285712f418c301ae5"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "06096db0b0bd36f8bb7b1fea96e5ef19"
server_key_iv: "91d2148032a3274632a07aace69c8028"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.659777 30291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:49.679041 30306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.680776 30291 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.003s
I20260620 21:57:49.680915 30291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "cebc303216474e8285712f418c301ae5"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "06096db0b0bd36f8bb7b1fea96e5ef19"
server_key_iv: "91d2148032a3274632a07aace69c8028"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:49.681059 30291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:49.721450 30291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:49.725167 30291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:49.725373 30291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:49.726091 30291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:49.727180 30291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:49.727253 30291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.727325 30291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:49.727358 30291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:49.739464 30291 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:36461
I20260620 21:57:49.740619 30291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:57:49.747390 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30291
I20260620 21:57:49.747509 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:57:49.747833 17679 external_mini_cluster.cc:1428] Setting key 2c23479a9a971cd2915135c0bccfc533
I20260620 21:57:49.743834 30419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:36461 every 8 connection(s)
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:49.772746 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.748240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:47807 (local address 127.17.67.254:41873)
0620 21:57:49.748644 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:49.748649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:49.758228 (+ 9579us) server_negotiation.cc:407] Connection header received
0620 21:57:49.759591 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:49.759595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:49.759677 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:49.759742 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:49.761585 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.762430 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.764245 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.764597 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.764714 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.768010 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:49.768014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:49.768017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:49.768048 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:49.769930 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.770707 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.770711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.770713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.770811 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:49.771263 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:49.771266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:49.771268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:49.771535 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:49.771625 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:49.772411 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 21:57:49.772553 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260620 21:57:49.774657 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:49.746079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.195:47807)
0620 21:57:49.757983 (+ 11904us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:49.758068 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:49.759285 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:49.760219 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:49.760230 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:49.760637 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:49.761385 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.761407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.762604 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.762619 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.764088 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:49.764100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:49.764934 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:49.764938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:49.765171 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:49.765982 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:49.766004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:49.767797 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:49.770099 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.770106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.770125 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.770524 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.770937 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:49.770940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:49.770943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:49.771145 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:49.771713 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:49.771720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:49.772089 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 21:57:49.772995 (+ 906us) client_negotiation.cc:240] Negotiation successful
0620 21:57:49.773400 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11598,"thread_start_us":148,"threads_started":1}
I20260620 21:57:49.775957 30420 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:49.776221 30420 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:49.776834 30420 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:49.777956 29936 ts_manager.cc:194] Registered new tserver with Master: cebc303216474e8285712f418c301ae5 (127.17.67.195:36461)
I20260620 21:57:49.778734 29936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:47807
I20260620 21:57:49.786456 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:57:49.787181 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30017
I20260620 21:57:49.794510 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30156
I20260620 21:57:49.801303 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30291
I20260620 21:57:49.808372 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 29904
I20260620 21:57:49.816385 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41873
--webserver_interface=127.17.67.254
--webserver_port=40457
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41873
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:49.941038 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:49.941308 30423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:49.941346 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:49.945286 30423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:49.945339 30423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:49.945359 30423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:49.945374 30423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:49.945390 30423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:49.951015 30423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41873
--ranger_config_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:49.952028 30423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:49.952996 30423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:49.961921 30428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.961948 30431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:49.961918 30429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:49.962396 30423 server_base.cc:1034] running on GCE node
I20260620 21:57:49.962898 30423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:49.964041 30423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:49.965216 30423 hybrid_clock.cc:648] HybridClock initialized: now 1781992669965183 us; error 43 us; skew 500 ppm
Jun 20 21:57:49 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:49.968067 30423 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:49.968920 30423 webserver.cc:460] Webserver started at http://127.17.67.254:40457/ using document root <none> and password file <none>
I20260620 21:57:49.969463 30423 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:49.969532 30423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992669, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:50.053449 30423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 21:57:50.056321 30439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.058032 30423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:57:50.058182 30423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "84c1327fb3344077bea2a84f55f97904"
format_stamp: "Formatted at 2026-06-20 21:57:47 on dist-test-slave-cx8d"
server_key: "9bb754a03b9b07eafb1c9ed8415240da"
server_key_iv: "a91d140a59194de1d105123ce98f9756"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:50.058915 30423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:50.090359 30423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:50.093118 30423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:50.093281 30423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:50.100435 30423 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41873
I20260620 21:57:50.100471 30491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41873 every 8 connection(s)
I20260620 21:57:50.101509 30423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:50.105808 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30423
I20260620 21:57:50.105895 30492 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:57:50.106352 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:34315
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:41873
--webserver_port=33089
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:57:50.108268 30492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Bootstrap starting.
I20260620 21:57:50.124955 30492 log.cc:824] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:50.131573 30492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:57:50.132079 30492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Bootstrap complete.
I20260620 21:57:50.136376 30492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:50.136762 30492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c1327fb3344077bea2a84f55f97904, State: Initialized, Role: FOLLOWER
I20260620 21:57:50.137295 30492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:50.137411 30492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:57:50.137503 30492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:57:50.137624 30492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:57:50.139516 30492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:50.139855 30492 leader_election.cc:302] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c1327fb3344077bea2a84f55f97904; no voters:
I20260620 21:57:50.140211 30492 leader_election.cc:288] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:57:50.140292 30496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:57:50.140622 30496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [term 2 LEADER]: Becoming Leader. State: Replica: 84c1327fb3344077bea2a84f55f97904, State: Running, Role: LEADER
I20260620 21:57:50.140975 30496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } }
I20260620 21:57:50.141458 30492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:50.142505 30497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84c1327fb3344077bea2a84f55f97904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } } }
I20260620 21:57:50.142524 30498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c1327fb3344077bea2a84f55f97904. Latest consensus state: current_term: 2 leader_uuid: "84c1327fb3344077bea2a84f55f97904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c1327fb3344077bea2a84f55f97904" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41873 } } }
I20260620 21:57:50.142637 30497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:50.142643 30498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904 [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:50.142966 30505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:50.146144 30505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:57:50.146626 30492 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:57:50.146806 30505 catalog_manager.cc:1259] Loaded cluster ID: 7bf8e6ac41af4ada883aece838641f98
I20260620 21:57:50.146852 30505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:50.149314 30505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:50.149971 30505 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Loaded TSK: 0
I20260620 21:57:50.151189 30505 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:57:50.246878 30494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:50.247144 30494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:50.247246 30494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:50.251611 30494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:50.251700 30494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:50.251807 30494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:57:50.259156 30494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:34315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=33089
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:50.260690 30494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:50.261966 30494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:50.272372 30532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:50.272370 30530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:50.272753 30529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:50.272850 30494 server_base.cc:1034] running on GCE node
I20260620 21:57:50.273439 30494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:50.274267 30494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:50.275458 30494 hybrid_clock.cc:648] HybridClock initialized: now 1781992670275419 us; error 53 us; skew 500 ppm
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:50.278836 30494 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:57:50.279881 30494 webserver.cc:460] Webserver started at http://127.17.67.193:33089/ using document root <none> and password file <none>
I20260620 21:57:50.280470 30494 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:50.280578 30494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:50.307778 30494 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 21:57:50.311177 30540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.312721 30494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:57:50.312839 30494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "a31c092f073947478797fc24f5fe81ab"
format_stamp: "Formatted at 2026-06-20 21:57:48 on dist-test-slave-cx8d"
server_key: "378beb3978e265492aa4f71b8043e9cc"
server_key_iv: "f69be5e372763ae088677f9b31123421"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:50.313598 30494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:50.335924 30494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:50.339478 30494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:50.339710 30494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:50.340463 30494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:50.341588 30494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:50.341655 30494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.341879 30494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:50.341970 30494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.356282 30494 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:34315
I20260620 21:57:50.357858 30494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:57:50.358469 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30494
I20260620 21:57:50.358994 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:46555
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:41873
--webserver_port=44939
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:57:50.359017 30653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:34315 every 8 connection(s)
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:50.391652 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.361700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.193:46551)
0620 21:57:50.363029 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:50.363097 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:50.364664 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:50.366866 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:50.366876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:50.367280 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:50.367964 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.367988 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.370186 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.370190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.371730 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.371743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.373140 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.373144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.373348 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.374389 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:50.374413 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:50.376327 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:50.387493 (+ 11166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.387501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.387526 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.387942 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.388466 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.388470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.388473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.388686 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.389648 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:50.389655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:50.389972 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 21:57:50.390190 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 21:57:50.390565 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":151,"threads_started":1}
I20260620 21:57:50.393649 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.361688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:46551 (local address 127.17.67.254:41873)
0620 21:57:50.363524 (+ 1836us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:50.363544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:50.363585 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:57:50.365344 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:50.365361 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:50.365890 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:50.366305 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:50.368201 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.370006 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.371892 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.372719 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.372892 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.376604 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:50.376612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:50.376633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:50.376671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:50.387296 (+ 10625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.388136 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.388147 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.388155 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.388330 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.388822 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.388825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.388827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.389392 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:50.389534 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:50.392241 (+ 2707us) server_negotiation.cc:299] Negotiation successful
0620 21:57:50.392585 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":122,"threads_started":1}
I20260620 21:57:50.397493 30654 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:50.397779 30654 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.398360 30654 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
W20260620 21:57:50.400698 30654 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:41873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:57:50.400877 30654 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.401094 30654 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:50.402051 30654 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.402242 30654 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
W20260620 21:57:50.403121 30654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:57:50.530917 30656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:50.531332 30656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:50.531421 30656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:50.536120 30656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:50.536196 30656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:50.536290 30656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:57:50.544418 30656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:46555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=44939
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:50.545958 30656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:50.547436 30656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:50.558101 30664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:50.558332 30656 server_base.cc:1034] running on GCE node
W20260620 21:57:50.558092 30663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:50.558092 30666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:50.559092 30656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:50.560009 30656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:50.561228 30656 hybrid_clock.cc:648] HybridClock initialized: now 1781992670561203 us; error 43 us; skew 500 ppm
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:50.565321 30656 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:57:50.566599 30656 webserver.cc:460] Webserver started at http://127.17.67.194:44939/ using document root <none> and password file <none>
I20260620 21:57:50.567340 30656 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:50.567447 30656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:50.591274 30656 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260620 21:57:50.594584 30674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.596230 30656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:57:50.596370 30656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "5af62a614d4b4e878e5004099882b106"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "ac24ff425d3ff55e7c2ea736167df294"
server_key_iv: "dfd1af2c0ea8187ffb0b3c39520e0601"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:50.597188 30656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:50.624567 30656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:50.628461 30656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:50.628683 30656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:50.629495 30656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:50.630628 30656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:50.630712 30656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.630779 30656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:50.630812 30656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.649210 30656 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:46555
I20260620 21:57:50.650195 30787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:46555 every 8 connection(s)
I20260620 21:57:50.650650 30656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:57:50.658890 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30656
I20260620 21:57:50.659405 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.67.212:42453/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:36461
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:41873
--webserver_port=39897
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:50.673480 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.653219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.194:43671)
0620 21:57:50.653858 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:50.653926 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:50.655278 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:50.655949 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:50.655959 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:50.656337 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:50.657079 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.657100 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.658312 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.658316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.660228 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.660241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.661137 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.661142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.661352 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.662321 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:50.662344 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:50.664212 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:50.669726 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.669734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.669754 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.670187 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.670554 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.670558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.670561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.670807 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.671358 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:50.671366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:50.671741 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 21:57:50.671964 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 21:57:50.672336 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":144,"threads_started":1}
I20260620 21:57:50.674005 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.652696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:43671 (local address 127.17.67.254:41873)
0620 21:57:50.652863 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:50.652868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:50.654043 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 21:57:50.655545 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:50.655549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:50.655613 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:50.655716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:50.657267 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.658143 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.660392 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.660819 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.660909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.664427 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:50.664431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:50.664434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:50.664466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:50.669554 (+ 5088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.670318 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.670323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.670325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.670427 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.670938 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.670942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.670944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.671195 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:50.671298 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:50.673694 (+ 2396us) server_negotiation.cc:299] Negotiation successful
0620 21:57:50.673840 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:57:50.675596 30788 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:50.675859 30788 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.676443 30788 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
W20260620 21:57:50.678077 30788 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:41873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:57:50.678205 30788 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.678371 30788 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:50.679152 30788 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.679327 30788 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
W20260620 21:57:50.680064 30788 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:57:50.807601 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:50.807909 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:50.807984 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:50.814056 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:50.814177 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:50.814288 30791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:57:50.820936 30791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.67.212:42453/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:36461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=39897
--tserver_master_addrs=127.17.67.254:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:50.822408 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:50.823776 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:50.836009 30797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:50.836009 30798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:50.837014 30800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:50.837545 30791 server_base.cc:1034] running on GCE node
I20260620 21:57:50.838115 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:50.838963 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:50.840173 30791 hybrid_clock.cc:648] HybridClock initialized: now 1781992670840152 us; error 37 us; skew 500 ppm
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:50.843739 30791 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:57:50.844702 30791 webserver.cc:460] Webserver started at http://127.17.67.195:39897/ using document root <none> and password file <none>
I20260620 21:57:50.845288 30791 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:50.845422 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:50.867134 30791 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:57:50.870347 30808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.871763 30791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:57:50.871876 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "cebc303216474e8285712f418c301ae5"
format_stamp: "Formatted at 2026-06-20 21:57:49 on dist-test-slave-cx8d"
server_key: "06096db0b0bd36f8bb7b1fea96e5ef19"
server_key_iv: "91d2148032a3274632a07aace69c8028"
server_key_version: "kuduclusterkey@0"
I20260620 21:57:50.872488 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:50.898432 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:50.902027 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:50.902246 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:50.903048 30791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:50.904336 30791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:50.904389 30791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.904444 30791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:50.904469 30791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:50.916608 30791 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:36461
I20260620 21:57:50.916636 30921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:36461 every 8 connection(s)
I20260620 21:57:50.917831 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:57:50.921639 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 30791
Jun 20 21:57:50 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992670, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:50.940002 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.922079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:51663 (local address 127.17.67.254:41873)
0620 21:57:50.922736 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:50.922740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:50.926760 (+ 4020us) server_negotiation.cc:407] Connection header received
0620 21:57:50.928019 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:50.928022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:50.928071 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:50.928140 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:50.929791 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.930541 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.932231 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.932579 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.932655 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.935714 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:50.935719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:50.935721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:50.935747 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:50.937556 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.938235 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.938239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.938241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.938323 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.938721 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.938726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.938729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.938948 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:50.939031 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:50.939701 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 21:57:50.939825 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560}
I20260620 21:57:50.941385 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.920321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41873 (local address 127.17.67.195:51663)
0620 21:57:50.926604 (+ 6283us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:50.926666 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:50.927767 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:50.928558 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:50.928567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:50.928915 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:50.929629 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.929650 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.930683 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.930687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.932077 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:50.932088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.932847 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.932850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:50.933044 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.933781 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:50.933801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:50.935509 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:50.937724 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.937731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.937749 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.938115 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.938424 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:50.938426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:50.938429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:50.938607 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:50.939110 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:50.939117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:50.939434 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 21:57:50.939643 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 21:57:50.939971 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6051,"thread_start_us":121,"threads_started":1}
I20260620 21:57:50.942691 30922 heartbeater.cc:347] Connected to a master server at 127.17.67.254:41873
I20260620 21:57:50.942939 30922 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.943507 30922 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
W20260620 21:57:50.945115 30922 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:41873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:57:50.945257 30922 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.945465 30922 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:50.946317 30922 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:50.946492 30922 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:50.946978 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:50.922895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.17.67.254:41873)
0620 21:57:50.923167 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:50.923189 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:50.924417 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 21:57:50.924477 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:50.924480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:50.924539 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:50.924658 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:50.927757 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.928567 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.940471 (+ 11904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:50.940870 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:50.940998 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:50.943098 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:50.943102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:50.943105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:50.943129 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:50.944997 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.945589 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.945594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.945596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.945702 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:50.946023 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:50.946026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:50.946028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:50.946259 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:50.946575 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:50.946668 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 21:57:50.946795 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
W20260620 21:57:50.947409 30922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 21:57:51 dist-test-slave-cx8d krb5kdc[29460](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992671, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:51.403374 30654 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:51.403821 30654 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:51.405239 30455 ts_manager.cc:194] Registered new tserver with Master: a31c092f073947478797fc24f5fe81ab (127.17.67.193:34315)
I20260620 21:57:51.406648 30455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:46551
I20260620 21:57:51.680323 30788 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:51.680631 30788 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:51.681457 30455 ts_manager.cc:194] Registered new tserver with Master: 5af62a614d4b4e878e5004099882b106 (127.17.67.194:46555)
I20260620 21:57:51.682361 30455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:43671
Jun 20 21:57:51 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992671, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:51.947669 30922 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:51.948170 30922 heartbeater.cc:511] Master 127.17.67.254:41873 requested a full tablet report, sending...
I20260620 21:57:51.949070 30455 ts_manager.cc:194] Registered new tserver with Master: cebc303216474e8285712f418c301ae5 (127.17.67.195:36461)
I20260620 21:57:51.949992 30455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:51663
I20260620 21:57:51.954560 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:57:51.971472 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:51.956355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.17.67.254:41873)
0620 21:57:51.956975 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:51.956980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:51.957356 (+ 376us) server_negotiation.cc:407] Connection header received
0620 21:57:51.957403 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:51.957406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:51.957454 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:51.957535 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:51.958539 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:51.959363 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:51.960513 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:51.960818 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:51.960912 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:51.963509 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:51.963514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:51.963516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:51.963543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:51.965465 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:51.970087 (+ 4622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:51.970091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:51.970093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:51.970183 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:51.970463 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:51.970467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:51.970468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:51.970695 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:51.970809 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:51.971161 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 21:57:51.971301 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":104,"threads_started":1}
Jun 20 21:57:51 dist-test-slave-cx8d krb5kdc[29460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992671, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for HTTP/127.17.67.212@KRBTEST.COM
I20260620 21:57:51.979547 30455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:57:51.985096 30455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:57:52.009814 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.001747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:46555 (local address 127.0.0.1:37312)
0620 21:57:52.003721 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.003735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:52.003871 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:52.004989 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:52.004992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:52.005009 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:52.005108 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.005114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.006434 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.006437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.007785 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.007798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.009195 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.009199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.009333 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.009377 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 21:57:52.009499 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:57:52.009654 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1915,"thread_start_us":91,"threads_started":1}
I20260620 21:57:52.010491 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.002133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.17.67.194:46555)
0620 21:57:52.004421 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.004430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.004454 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:57:52.004525 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.004529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.004706 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.004835 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:52.005233 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.006325 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.007955 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.008824 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.008934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.009126 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.010242 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.010353 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2161,"thread_start_us":116,"threads_started":1}
I20260620 21:57:52.013903 30722 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 (DEFAULT_TABLE table=test-table [id=0f66e9dbd65e45e3b6c4920a17e30aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:52.015126 30722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ca2891a304e21830eacdd5349b2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:52.024622 30957 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Bootstrap starting.
I20260620 21:57:52.027856 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.000460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:34315 (local address 127.0.0.1:40080)
0620 21:57:52.003384 (+ 2924us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.003440 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:52.003871 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:52.004680 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:52.004687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:52.004710 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:52.004858 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.004866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.006382 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.006386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.025956 (+ 19570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.025970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.027386 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.027390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.027505 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.027530 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:57:52.027606 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:57:52.027690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2737,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260620 21:57:52.028666 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.000607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.17.67.193:34315)
0620 21:57:52.003771 (+ 3164us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.003780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.004011 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:57:52.004100 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.004107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.004345 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.004507 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:52.004989 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.006247 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.026132 (+ 19885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.027035 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.027159 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.027312 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.028397 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.028522 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3049,"thread_start_us":61,"threads_started":1}
I20260620 21:57:52.028759 30957 tablet_bootstrap.cc:654] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:52.029697 30957 log.cc:824] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:52.032137 30575 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 (DEFAULT_TABLE table=test-table [id=0f66e9dbd65e45e3b6c4920a17e30aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:52.033293 30575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ca2891a304e21830eacdd5349b2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:52.034441 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.002891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:36461 (local address 127.0.0.1:36586)
0620 21:57:52.004099 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.004111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:52.004181 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:52.022184 (+ 18003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:52.022188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:52.022212 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:52.022326 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.022332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.031142 (+ 8810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.031146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.032398 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.032412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.034059 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.034063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.034155 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.034183 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:57:52.034231 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 21:57:52.034306 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":116,"threads_started":1}
I20260620 21:57:52.035157 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.003059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.17.67.195:36461)
0620 21:57:52.021700 (+ 18641us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.021708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.021736 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:57:52.021819 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.021825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.022036 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.022205 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:52.024815 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.030980 (+ 6165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.032858 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.033915 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.034731 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.034885 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.034937 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.035026 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18490,"thread_start_us":94,"threads_started":1}
I20260620 21:57:52.038130 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 (DEFAULT_TABLE table=test-table [id=0f66e9dbd65e45e3b6c4920a17e30aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:52.039175 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ca2891a304e21830eacdd5349b2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:52.053259 30960 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Bootstrap starting.
I20260620 21:57:52.053890 30959 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Bootstrap starting.
I20260620 21:57:52.054260 30957 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: No bootstrap required, opened a new log
I20260620 21:57:52.054621 30957 ts_tablet_manager.cc:1397] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20260620 21:57:52.056460 30959 tablet_bootstrap.cc:654] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:52.059489 30960 tablet_bootstrap.cc:654] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:52.062122 30960 log.cc:824] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:52.064661 30957 raft_consensus.cc:348] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.065217 30957 raft_consensus.cc:374] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:52.065330 30957 raft_consensus.cc:729] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af62a614d4b4e878e5004099882b106, State: Initialized, Role: FOLLOWER
I20260620 21:57:52.065953 30957 consensus_queue.cc:260] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.066491 30959 log.cc:824] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:52.067147 30957 ts_tablet_manager.cc:1428] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260620 21:57:52.067549 30788 heartbeater.cc:503] Master 127.17.67.254:41873 was elected leader, sending a full tablet report...
I20260620 21:57:52.069049 30959 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: No bootstrap required, opened a new log
I20260620 21:57:52.069283 30959 ts_tablet_manager.cc:1397] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260620 21:57:52.071310 30960 tablet_bootstrap.cc:492] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: No bootstrap required, opened a new log
I20260620 21:57:52.071533 30960 ts_tablet_manager.cc:1397] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260620 21:57:52.073423 30959 raft_consensus.cc:348] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.073930 30959 raft_consensus.cc:374] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:52.074040 30959 raft_consensus.cc:729] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31c092f073947478797fc24f5fe81ab, State: Initialized, Role: FOLLOWER
I20260620 21:57:52.074625 30959 consensus_queue.cc:260] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.075806 30959 ts_tablet_manager.cc:1428] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:57:52.076226 30654 heartbeater.cc:503] Master 127.17.67.254:41873 was elected leader, sending a full tablet report...
I20260620 21:57:52.075875 30960 raft_consensus.cc:348] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.077319 30960 raft_consensus.cc:374] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:52.077396 30960 raft_consensus.cc:729] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebc303216474e8285712f418c301ae5, State: Initialized, Role: FOLLOWER
I20260620 21:57:52.087277 30960 consensus_queue.cc:260] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.088867 30960 ts_tablet_manager.cc:1428] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260620 21:57:52.089411 30922 heartbeater.cc:503] Master 127.17.67.254:41873 was elected leader, sending a full tablet report...
W20260620 21:57:52.112434 30655 tablet.cc:2369] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:52.119093 30962 raft_consensus.cc:482] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:57:52.119333 30962 raft_consensus.cc:504] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.120818 30962 leader_election.cc:288] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a31c092f073947478797fc24f5fe81ab (127.17.67.193:34315), cebc303216474e8285712f418c301ae5 (127.17.67.195:36461)
I20260620 21:57:52.131379 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.121844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:43183 (local address 127.17.67.193:34315)
0620 21:57:52.121994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.121999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.122974 (+ 975us) server_negotiation.cc:407] Connection header received
0620 21:57:52.123217 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.123220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.123271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.123325 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:52.123803 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.124666 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.129975 (+ 5309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.130637 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.130739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.130796 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.131153 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.131234 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:57:52.132085 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.121328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:34315 (local address 127.17.67.194:43183)
0620 21:57:52.122876 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.122891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:52.123074 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:52.123493 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:52.123496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:52.123517 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:52.123666 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.123672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.124822 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.124826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.126302 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.126313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.130895 (+ 4582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.130899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.131009 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.131032 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:57:52.131832 (+ 800us) client_negotiation.cc:240] Negotiation successful
0620 21:57:52.131914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"thread_start_us":148,"threads_started":1}
I20260620 21:57:52.132993 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ca2891a304e21830eacdd5349b2c2" candidate_uuid: "5af62a614d4b4e878e5004099882b106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31c092f073947478797fc24f5fe81ab" is_pre_election: true
I20260620 21:57:52.133342 30608 raft_consensus.cc:2454] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5af62a614d4b4e878e5004099882b106 in term 0.
I20260620 21:57:52.133980 30676 leader_election.cc:302] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af62a614d4b4e878e5004099882b106, a31c092f073947478797fc24f5fe81ab; no voters:
I20260620 21:57:52.134301 30962 raft_consensus.cc:2781] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:57:52.134374 30962 raft_consensus.cc:482] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:57:52.134415 30962 raft_consensus.cc:3037] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:52.136250 30962 raft_consensus.cc:504] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.136684 30962 leader_election.cc:288] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [CANDIDATE]: Term 1 election: Requested vote from peers a31c092f073947478797fc24f5fe81ab (127.17.67.193:34315), cebc303216474e8285712f418c301ae5 (127.17.67.195:36461)
I20260620 21:57:52.137321 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ca2891a304e21830eacdd5349b2c2" candidate_uuid: "5af62a614d4b4e878e5004099882b106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31c092f073947478797fc24f5fe81ab"
I20260620 21:57:52.137445 30608 raft_consensus.cc:3037] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:52.139029 30608 raft_consensus.cc:2454] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af62a614d4b4e878e5004099882b106 in term 1.
I20260620 21:57:52.139415 30676 leader_election.cc:302] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af62a614d4b4e878e5004099882b106, a31c092f073947478797fc24f5fe81ab; no voters:
I20260620 21:57:52.139649 30962 raft_consensus.cc:2781] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:52.139866 30962 raft_consensus.cc:686] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 1 LEADER]: Becoming Leader. State: Replica: 5af62a614d4b4e878e5004099882b106, State: Running, Role: LEADER
I20260620 21:57:52.140235 30962 consensus_queue.cc:237] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } }
I20260620 21:57:52.144290 30455 catalog_manager.cc:5495] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5af62a614d4b4e878e5004099882b106 (127.17.67.194). New cstate: current_term: 1 leader_uuid: "5af62a614d4b4e878e5004099882b106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af62a614d4b4e878e5004099882b106" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 46555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 } health_report { overall_health: UNKNOWN } } }
I20260620 21:57:52.147156 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.122660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54505 (local address 127.17.67.195:36461)
0620 21:57:52.122802 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.122807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.126672 (+ 3865us) server_negotiation.cc:407] Connection header received
0620 21:57:52.126864 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.126868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.126920 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.126970 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:52.127360 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.128225 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.132772 (+ 4547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.133373 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.133490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.133545 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.146889 (+ 13344us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.147011 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:57:52.148191 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.122252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:36461 (local address 127.17.67.194:54505)
0620 21:57:52.126582 (+ 4330us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.126597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:52.126743 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:52.127102 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:52.127105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:52.127122 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:52.127235 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.127240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.128379 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.128382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.129820 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:52.129832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.146352 (+ 16520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.146357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:52.146683 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.146718 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:57:52.147935 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0620 21:57:52.148034 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4261,"thread_start_us":101,"threads_started":1}
I20260620 21:57:52.149132 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ca2891a304e21830eacdd5349b2c2" candidate_uuid: "5af62a614d4b4e878e5004099882b106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebc303216474e8285712f418c301ae5"
I20260620 21:57:52.149132 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ca2891a304e21830eacdd5349b2c2" candidate_uuid: "5af62a614d4b4e878e5004099882b106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebc303216474e8285712f418c301ae5" is_pre_election: true
I20260620 21:57:52.149405 30858 raft_consensus.cc:3037] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:52.150832 30858 raft_consensus.cc:2454] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5af62a614d4b4e878e5004099882b106 in term 1.
W20260620 21:57:52.155740 30789 tablet.cc:2369] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:52.167232 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.160065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.17.67.194:46555)
0620 21:57:52.160876 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:52.160880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:52.160896 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:57:52.160972 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:52.160975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:52.161039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:52.161161 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:57:52.161548 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.162394 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.164664 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:52.165059 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:52.165187 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:52.165722 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:57:52.166361 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:57:52.166497 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:52.166988 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 21:57:52.167058 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
W20260620 21:57:52.169662 30923 tablet.cc:2369] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:52.176873 30858 raft_consensus.cc:1261] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 1 FOLLOWER]: Refusing update from remote peer 5af62a614d4b4e878e5004099882b106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:52.177011 30608 raft_consensus.cc:1261] T 2f3ca2891a304e21830eacdd5349b2c2 P a31c092f073947478797fc24f5fe81ab [term 1 FOLLOWER]: Refusing update from remote peer 5af62a614d4b4e878e5004099882b106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:52.177814 30968 consensus_queue.cc:1035] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a31c092f073947478797fc24f5fe81ab" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:52.178073 30962 consensus_queue.cc:1035] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cebc303216474e8285712f418c301ae5" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 36461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:52.186236 30971 mvcc.cc:205] Tried to move back new op lower bound from 7299041985227280384 to 7299041985090289664. Current Snapshot: MvccSnapshot[applied={T|T < 7299041985227280384}]
I20260620 21:57:52.193035 30972 mvcc.cc:205] Tried to move back new op lower bound from 7299041985227280384 to 7299041985090289664. Current Snapshot: MvccSnapshot[applied={T|T < 7299041985227280384}]
I20260620 21:57:52.205240 30973 mvcc.cc:205] Tried to move back new op lower bound from 7299041985227280384 to 7299041985090289664. Current Snapshot: MvccSnapshot[applied={T|T < 7299041985227280384}]
I20260620 21:57:52.206211 30455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:57:52.206415 30455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:57:52.209465 30455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84c1327fb3344077bea2a84f55f97904: Sending DeleteTablet for 3 replicas of tablet 2f3ca2891a304e21830eacdd5349b2c2
I20260620 21:57:52.210580 30853 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:52 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:36586
I20260620 21:57:52.210742 30575 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:52 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:40080
I20260620 21:57:52.211464 30722 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ca2891a304e21830eacdd5349b2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:57:52 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:37312
I20260620 21:57:52.212780 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30494
I20260620 21:57:52.216985 30982 tablet_replica.cc:330] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: stopping tablet replica
I20260620 21:57:52.217474 30982 raft_consensus.cc:2229] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:57:52.217969 30982 raft_consensus.cc:2258] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:57:52.218469 30980 tablet_replica.cc:330] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: stopping tablet replica
I20260620 21:57:52.218807 30980 raft_consensus.cc:2229] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:57:52.219102 30980 raft_consensus.cc:2258] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:57:52.220705 30982 ts_tablet_manager.cc:1905] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:57:52.223704 30982 ts_tablet_manager.cc:1918] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:57:52.223814 30982 log.cc:1196] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/2f3ca2891a304e21830eacdd5349b2c2
W20260620 21:57:52.223892 30441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:34315 (error 108)
I20260620 21:57:52.224200 30982 ts_tablet_manager.cc:1939] T 2f3ca2891a304e21830eacdd5349b2c2 P 5af62a614d4b4e878e5004099882b106: Deleting consensus metadata
I20260620 21:57:52.225502 30440 catalog_manager.cc:4841] TS 5af62a614d4b4e878e5004099882b106 (127.17.67.194:46555): tablet 2f3ca2891a304e21830eacdd5349b2c2 (table test-table [id=0f66e9dbd65e45e3b6c4920a17e30aad]) successfully deleted
I20260620 21:57:52.226717 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30656
I20260620 21:57:52.229960 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:52.229604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:34315 (local address 127.0.0.1:40092)
0620 21:57:52.229796 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:52.229872 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:34315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 21:57:52.230206 30441 catalog_manager.cc:4584] TS a31c092f073947478797fc24f5fe81ab (127.17.67.193:34315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f3ca2891a304e21830eacdd5349b2c2: Network error: Client connection negotiation failed: client connection to 127.17.67.193:34315: connect: Connection refused (error 111)
I20260620 21:57:52.231298 30980 ts_tablet_manager.cc:1905] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:57:52.234947 30980 ts_tablet_manager.cc:1918] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:57:52.235085 30980 log.cc:1196] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/2f3ca2891a304e21830eacdd5349b2c2
I20260620 21:57:52.235450 30980 ts_tablet_manager.cc:1939] T 2f3ca2891a304e21830eacdd5349b2c2 P cebc303216474e8285712f418c301ae5: Deleting consensus metadata
I20260620 21:57:52.236900 30443 catalog_manager.cc:4841] TS cebc303216474e8285712f418c301ae5 (127.17.67.195:36461): tablet 2f3ca2891a304e21830eacdd5349b2c2 (table test-table [id=0f66e9dbd65e45e3b6c4920a17e30aad]) successfully deleted
I20260620 21:57:52.241302 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30791
I20260620 21:57:52.247967 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 30423
I20260620 21:57:52.254002 17679 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:57:52.589172 17679 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:57:52.589283 17679 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:57:52.934532 17679 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:57:52.934 UTC [29490] LOG: received smart shutdown request
2026-06-20 21:57:52.938 UTC [29490] LOG: background worker "logical replication launcher" (PID 29499) exited with exit code 1
2026-06-20 21:57:52.938 UTC [29494] LOG: shutting down
2026-06-20 21:57:53.101 UTC [29490] LOG: database system is shut down
2026-06-20T21:57:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49584 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:57:53 dist-test-slave-cx8d krb5kdc[30992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:57:53 dist-test-slave-cx8d krb5kdc[30992](info): set up 2 sockets
Jun 20 21:57:53 dist-test-slave-cx8d krb5kdc[30992](info): commencing operation
krb5kdc: starting...
W20260620 21:57:54.285492 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:57:54.404351 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38711
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:54.512590 31005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:54.512916 31005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:54.512956 31005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:54.517036 31005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:54.517097 31005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:54.517117 31005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:54.517133 31005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:54.517148 31005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:54.522629 31005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:54.523725 31005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:54.524760 31005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:54.532928 31013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:54.532966 31011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:54.532979 31010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:54.533176 31005 server_base.cc:1034] running on GCE node
I20260620 21:57:54.533807 31005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:54.534816 31005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:54.535988 31005 hybrid_clock.cc:648] HybridClock initialized: now 1781992674535960 us; error 30 us; skew 500 ppm
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:54.538681 31005 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:54.539522 31005 webserver.cc:460] Webserver started at http://127.17.67.254:33791/ using document root <none> and password file <none>
I20260620 21:57:54.540060 31005 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:54.540146 31005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:54.540372 31005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:54.542390 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "04e47ebb840e4562812f19dc2f93c49d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "80582abac247dd74ef4f237e3cd0a501"
server_key_iv: "d044437dbd5ccaf20b910f64947ac67a"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.542950 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "04e47ebb840e4562812f19dc2f93c49d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "80582abac247dd74ef4f237e3cd0a501"
server_key_iv: "d044437dbd5ccaf20b910f64947ac67a"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.546561 31005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:57:54.548946 31020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:54.550348 31005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:57:54.550485 31005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "04e47ebb840e4562812f19dc2f93c49d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "80582abac247dd74ef4f237e3cd0a501"
server_key_iv: "d044437dbd5ccaf20b910f64947ac67a"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.550598 31005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:54.563019 31005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:54.566012 31005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:54.566211 31005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:54.573135 31005 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:38711
I20260620 21:57:54.573170 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:38711 every 8 connection(s)
I20260620 21:57:54.574123 31005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:54.576809 31073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:54.580279 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31005
I20260620 21:57:54.580437 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:57:54.580736 17679 external_mini_cluster.cc:1428] Setting key aa720090e86df75ec565095416fa8f2b
I20260620 21:57:54.580605 31073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.67.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.253@KRBTEST.COM" created.
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:54.598232 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.583040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.17.67.254:38711)
0620 21:57:54.583485 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:54.583504 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:54.583822 (+ 318us) server_negotiation.cc:407] Connection header received
0620 21:57:54.585363 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:54.585368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:54.585619 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:54.585791 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:54.586996 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.587861 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.589392 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.589869 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.590003 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.593132 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:54.593142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:54.593159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:54.593191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:54.594928 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.595531 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.595536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.595537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.595683 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.596078 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.596081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.596082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.596437 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:54.596530 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:54.597082 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 21:57:54.597315 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260620 21:57:54.598234 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.583040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38711 (local address 127.0.0.1:39200)
0620 21:57:54.583636 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.583695 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:54.584941 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:54.585837 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:54.585846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:54.586368 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:54.586783 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.586807 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.587993 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.587998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.589267 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.589278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.590011 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.590014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.590115 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.590673 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:54.590689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:54.592979 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:54.595045 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.595057 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.595071 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.595419 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.595799 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.595805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.595810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.595968 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.596576 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:54.596587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:54.596773 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 21:57:54.596984 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 21:57:54.597442 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":78,"thread_start_us":138,"threads_started":1}
I20260620 21:57:54.600832 31073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:54.601611 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.601332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46436)
0620 21:57:54.601442 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.601509 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:57:54.601856 31021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
I20260620 21:57:54.602833 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.602601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46438)
0620 21:57:54.602731 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.602779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 21:57:54.602998 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:36257: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
I20260620 21:57:54.608422 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:36257
--webserver_interface=127.17.67.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:57:54.652329 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } attempt: 1
I20260620 21:57:54.653262 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.652933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46450)
0620 21:57:54.653097 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.653183 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 21:57:54.654049 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.653830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46464)
0620 21:57:54.653939 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.653993 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:57:54.654184 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:36257: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
W20260620 21:57:54.719049 31080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:54.719365 31080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:54.719444 31080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:57:54.722369 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } attempt: 2
I20260620 21:57:54.723167 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.722871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46480)
0620 21:57:54.723021 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.723095 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 21:57:54.723753 31080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:54.723827 31080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:54.723865 31080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:54.723920 31080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:54.723960 31080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:54.723978 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.723702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46492)
0620 21:57:54.723812 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.723887 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:57:54.724123 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:36257: Network error: Client connection negotiation failed: client connection to 127.17.67.253:36257: connect: Connection refused (error 111)
I20260620 21:57:54.729482 31080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.67.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:54.730581 31080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:54.731614 31080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:54.740290 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:54.740337 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:54.740290 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:54.740576 31080 server_base.cc:1034] running on GCE node
I20260620 21:57:54.741225 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:54.742736 31080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:54.743902 31080 hybrid_clock.cc:648] HybridClock initialized: now 1781992674743889 us; error 30 us; skew 500 ppm
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:54.746639 31080 init.cc:377] Logged in from keytab as kudu/127.17.67.253@KRBTEST.COM (short username kudu)
I20260620 21:57:54.747671 31080 webserver.cc:460] Webserver started at http://127.17.67.253:37329/ using document root <none> and password file <none>
I20260620 21:57:54.748246 31080 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:54.748338 31080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:54.748584 31080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:54.750741 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data/instance:
uuid: "36c9453786a0418cbb1ecf53c82cc83d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "9c986997b642799db57d9a5d3f62dc15"
server_key_iv: "a413d9d932f2240edab795d921ecad67"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.751309 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal/instance:
uuid: "36c9453786a0418cbb1ecf53c82cc83d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "9c986997b642799db57d9a5d3f62dc15"
server_key_iv: "a413d9d932f2240edab795d921ecad67"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.755059 31080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:57:54.757505 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:54.759227 31080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:57:54.759402 31080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "36c9453786a0418cbb1ecf53c82cc83d"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "9c986997b642799db57d9a5d3f62dc15"
server_key_iv: "a413d9d932f2240edab795d921ecad67"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.759542 31080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:54.789238 31080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:54.792368 31080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:54.792604 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:54.801028 31080 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.253:36257
I20260620 21:57:54.801047 31147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.253:36257 every 8 connection(s)
I20260620 21:57:54.802137 31080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
I20260620 21:57:54.804773 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31080
I20260620 21:57:54.804910 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-1/wal/instance
I20260620 21:57:54.805001 31148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:54.805243 17679 external_mini_cluster.cc:1428] Setting key b6b243bd9c6853b79f57b0771548f63f
I20260620 21:57:54.809196 31148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.67.252@KRBTEST.COM; defaulting to no policy
I20260620 21:57:54.815292 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.252@KRBTEST.COM" created.
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
I20260620 21:57:54.825925 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.811513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.17.67.254:38711)
0620 21:57:54.811678 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:54.811683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:54.812585 (+ 902us) server_negotiation.cc:407] Connection header received
0620 21:57:54.813612 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:54.813615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:54.813676 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:54.813759 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:54.815076 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.815792 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.817152 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.817688 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.817820 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.820730 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:54.820735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:54.820742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:54.820765 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:54.822874 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.823698 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.823705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.823710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.823804 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.824166 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.824171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.824172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.824347 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:54.824444 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:54.825610 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0620 21:57:54.825780 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":64}
I20260620 21:57:54.826380 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.811922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38711 (local address 127.0.0.1:39208)
0620 21:57:54.812399 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.812450 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:54.813362 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:54.813919 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:54.813926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:54.814341 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:54.814942 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.814959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.815931 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.815935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.817013 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.817026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.817858 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.817863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.818065 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.818736 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:54.818781 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:54.820597 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:54.823042 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.823056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.823071 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.823521 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.823922 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.823925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.823927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.824066 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.824459 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:54.824467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:54.824707 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 21:57:54.824920 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 21:57:54.825272 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":143,"threads_started":1}
I20260620 21:57:54.827033 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.815689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46502)
0620 21:57:54.815849 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.815869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:54.815966 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:54.816922 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:54.816925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:54.817114 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:54.817286 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.817292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.818820 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.818825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.819708 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.819719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.820498 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.820505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.820659 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.821164 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:54.821184 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:54.822681 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:54.825054 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.825061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.825066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.825335 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.825771 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.825775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.825779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.825884 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.826516 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:54.826523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:54.826646 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 21:57:54.826739 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:57:54.826889 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":47}
I20260620 21:57:54.828138 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.815973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.17.67.253:36257)
0620 21:57:54.816384 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:54.816399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:54.816426 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:57:54.816476 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:54.816485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:54.816735 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:54.816920 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:54.817446 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.818690 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.819832 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.820361 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.820468 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.822815 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:54.822821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:54.822834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:54.822858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:54.824888 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.825496 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.825503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.825505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.825635 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.826044 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.826050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.826053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.826370 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:54.827645 (+ 1275us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:54.827753 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 21:57:54.827918 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":168,"threads_started":1}
I20260620 21:57:54.828611 31148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:54.830142 31073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:54.830894 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.830624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36292)
0620 21:57:54.830758 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.830819 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 21:57:54.831696 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.831384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36306)
0620 21:57:54.831573 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.831630 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 21:57:54.831984 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
I20260620 21:57:54.836784 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:36691
--webserver_interface=127.17.67.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:57:54.839377 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.829179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46506)
0620 21:57:54.829344 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.829360 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:54.829434 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:54.829824 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:54.829827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:54.830006 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:54.830149 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.830157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.831292 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.831299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.832272 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:54.832283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.832831 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.832834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:54.833000 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.833546 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:54.833559 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:54.835080 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:54.837756 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.837762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.837767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.838046 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.838430 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:54.838432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:54.838434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:54.838509 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:54.839004 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:54.839006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:54.839082 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 21:57:54.839163 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 21:57:54.839260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":37}
I20260620 21:57:54.839653 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.829254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.17.67.253:36257)
0620 21:57:54.829547 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:54.829554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:54.829601 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:57:54.829651 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:54.829653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:54.829696 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:54.829767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:54.830292 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.831172 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.832398 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:54.832683 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:54.832823 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:54.835231 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:54.835237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:54.835239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:54.835266 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:54.837616 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.838187 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.838193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.838196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.838311 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:54.838606 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:54.838611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:54.838615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:54.838893 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:54.839024 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:54.839263 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 21:57:54.839407 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260620 21:57:54.840378 31148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:54.841096 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.840806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36312)
0620 21:57:54.840913 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.841017 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:57:54.841360 31099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
I20260620 21:57:54.842473 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.842282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36316)
0620 21:57:54.842388 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.842429 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 21:57:54.842753 31148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
I20260620 21:57:54.863179 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 1
I20260620 21:57:54.864143 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.863768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36332)
0620 21:57:54.863954 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.864031 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 21:57:54.864959 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.864733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36336)
0620 21:57:54.864845 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.864902 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:57:54.865104 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
I20260620 21:57:54.892104 31148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 1
I20260620 21:57:54.893010 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.892710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36342)
0620 21:57:54.892868 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.892930 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:57:54.893905 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.893698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36352)
0620 21:57:54.893801 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.893853 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:57:54.894037 31148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
I20260620 21:57:54.940280 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 2
I20260620 21:57:54.941170 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.940810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36356)
0620 21:57:54.941017 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.941077 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:57:54.941982 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.941742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36364)
0620 21:57:54.941858 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.941913 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:57:54.942111 31073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
W20260620 21:57:54.956869 31155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:54.957175 31155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:54.957283 31155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:54.961454 31155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:54.961551 31155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:54.961608 31155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:54.961652 31155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:54.961688 31155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:54.962239 31148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 2
I20260620 21:57:54.963141 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.962821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36370)
0620 21:57:54.962979 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.963062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 21:57:54.964007 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:54.963736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36378)
0620 21:57:54.963869 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:54.963943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:57:54.964147 31148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:36691: Network error: Client connection negotiation failed: client connection to 127.17.67.252:36691: connect: Connection refused (error 111)
I20260620 21:57:54.967514 31155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:38711,127.17.67.253:36257,127.17.67.252:36691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:36691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.67.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:54.968596 31155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:54.969640 31155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:54.978269 31162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:54.978430 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:54.978490 31155 server_base.cc:1034] running on GCE node
W20260620 21:57:54.978243 31161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:54.979103 31155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:54.980247 31155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:54.981441 31155 hybrid_clock.cc:648] HybridClock initialized: now 1781992674981399 us; error 55 us; skew 500 ppm
Jun 20 21:57:54 dist-test-slave-cx8d krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:54.984617 31155 init.cc:377] Logged in from keytab as kudu/127.17.67.252@KRBTEST.COM (short username kudu)
I20260620 21:57:54.985545 31155 webserver.cc:460] Webserver started at http://127.17.67.252:40643/ using document root <none> and password file <none>
I20260620 21:57:54.986171 31155 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:54.986264 31155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:54.986495 31155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:54.988524 31155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data/instance:
uuid: "b30f6f0683ad4bf79ed0afc917b72d70"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "aa8e7f590685f36c6f8ee32549539010"
server_key_iv: "a2bbb9b221952aca59b1de9be73cc9a4"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.989122 31155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal/instance:
uuid: "b30f6f0683ad4bf79ed0afc917b72d70"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "aa8e7f590685f36c6f8ee32549539010"
server_key_iv: "a2bbb9b221952aca59b1de9be73cc9a4"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.992952 31155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 21:57:54.995859 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:54.997227 31155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:57:54.997493 31155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "b30f6f0683ad4bf79ed0afc917b72d70"
format_stamp: "Formatted at 2026-06-20 21:57:54 on dist-test-slave-cx8d"
server_key: "aa8e7f590685f36c6f8ee32549539010"
server_key_iv: "a2bbb9b221952aca59b1de9be73cc9a4"
server_key_version: "encryptionkey@0"
I20260620 21:57:54.997684 31155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:55.022368 31155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:55.026074 31155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:55.026312 31155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:55.033748 31155 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.252:36691
I20260620 21:57:55.033780 31223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.252:36691 every 8 connection(s)
I20260620 21:57:55.034713 31155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
I20260620 21:57:55.037472 31224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:55.041291 31073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 3
I20260620 21:57:55.041528 31224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:55.044376 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31155
I20260620 21:57:55.044515 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992513246304-17679-0/minicluster-data/master-2/wal/instance
I20260620 21:57:55.044857 17679 external_mini_cluster.cc:1428] Setting key 80a455732cafd94645a4c90f6379ba3a
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:57:55.055320 31148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } attempt: 3
I20260620 21:57:55.057403 31077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.041887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36388)
0620 21:57:55.042065 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:55.042083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:55.042183 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:55.044763 (+ 2580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:55.044766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:55.044973 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:55.045203 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.045212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.047185 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.047188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.048006 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.048016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.048874 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.048988 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.049849 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:55.049861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:55.052121 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:55.055198 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.055203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.055205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.055465 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.055878 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.055881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.055883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.056001 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.056839 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:55.056842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:55.056974 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 21:57:55.057065 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:57:55.057218 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 21:57:55.058152 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.043776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.17.67.254:38711)
0620 21:57:55.043916 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.043923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.044317 (+ 394us) server_negotiation.cc:407] Connection header received
0620 21:57:55.044766 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.044769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.044819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.044872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.046092 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.046826 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048329 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.048665 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048760 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.052964 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.052967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.052970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.052995 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.054802 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.055773 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.055777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.055779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.055862 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.056306 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.056309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.056311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.056617 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.056690 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.057216 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.057343 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:57:55.058948 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.045540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.17.67.252:36691)
0620 21:57:55.045856 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.045865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.046003 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:57:55.046129 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.046134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.046210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.046308 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.046991 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.047624 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048642 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.049134 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.049289 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.053606 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.053614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.053620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.053651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.056188 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.056672 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.056675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.056677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.056760 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.057049 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.057052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.057056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.057248 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.057425 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.058136 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.058274 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":257,"thread_start_us":97,"threads_started":1}
I20260620 21:57:55.058948 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.043707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38711 (local address 127.0.0.1:39220)
0620 21:57:55.044166 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:55.044221 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:55.044509 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:55.045086 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:55.045092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:55.045592 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:55.045924 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.045944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.046947 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.046952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.048205 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.048212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.048817 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.049074 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.050225 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:55.050247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:55.052774 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:55.054949 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.054960 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.054972 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.055639 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.055985 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.055990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.055994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.056134 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.056767 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:55.056774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:55.056938 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 21:57:55.057450 (+ 512us) client_negotiation.cc:240] Negotiation successful
0620 21:57:55.057839 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":103,"thread_start_us":159,"threads_started":1}
I20260620 21:57:55.059633 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.042691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.17.67.252:36691)
0620 21:57:55.043145 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.043159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.043192 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:57:55.044112 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.044135 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.044542 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.044950 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.045499 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.047065 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.048132 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.048742 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.049237 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.052327 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.052335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.052350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.052378 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.055086 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.055603 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.055609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.055611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.055760 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.056129 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.056134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.056138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.056701 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.058831 (+ 2130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.058909 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.059458 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":318,"spinlock_wait_cycles":4864,"thread_start_us":139,"threads_started":1}
I20260620 21:57:55.061216 31224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:57:55.064299 31073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: Bootstrap starting.
I20260620 21:57:55.066891 31073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: Neither blocks nor log segments found. Creating new log.
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
I20260620 21:57:55.067811 31073 log.cc:824] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:55.069060 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.055818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36408)
0620 21:57:55.056128 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:55.056145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:55.056229 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:55.057016 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:55.057019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:55.057218 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:55.057332 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.057340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.058684 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.058692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.061102 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.061113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.062168 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.062173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.062268 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.062812 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:55.062829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:55.064429 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:55.067078 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.067085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.067088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.067309 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.067697 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.067701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.067703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.067817 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.068563 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:55.068567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:55.068660 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 21:57:55.068754 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:57:55.068892 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260620 21:57:55.069612 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.056372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.17.67.252:36691)
0620 21:57:55.056750 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.056762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.056778 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:57:55.056818 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.056823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.056866 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.057468 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.057623 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.058539 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.061674 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.062030 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.062153 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.064642 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.064646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.064649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.064673 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.066955 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.067415 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.067421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.067423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.067602 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.067959 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.067962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.067964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.068444 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.069219 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.069293 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.069399 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260620 21:57:55.070204 31073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: No bootstrap required, opened a new log
I20260620 21:57:55.071593 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.061733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:36257 (local address 127.0.0.1:46520)
0620 21:57:55.062386 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:55.062400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:55.062510 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:55.062834 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:55.062837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:55.063010 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:55.063140 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.063149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.064077 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.064084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.064991 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.065002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.065518 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.065522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.065663 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.066087 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:55.066100 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:55.067738 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:55.069953 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.069957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.069959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.070196 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.070529 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.070535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.070537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.070632 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.071129 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:55.071132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:55.071230 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 21:57:55.071337 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:57:55.071454 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593}
I20260620 21:57:55.071694 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.061986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.17.67.253:36257)
0620 21:57:55.062232 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.062236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.062452 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:57:55.062607 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.062610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.062655 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.062706 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.063274 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.063954 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.065125 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.065393 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.065520 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.067874 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.067881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.067887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.067913 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.069845 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.070327 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.070330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.070332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.070420 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.070743 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.070750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.070755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.071016 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.071121 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.071413 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.071547 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 21:57:55.072397 31224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:57:55.072805 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: Bootstrap starting.
I20260620 21:57:55.074433 31073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.074676 31073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:55.074751 31073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e47ebb840e4562812f19dc2f93c49d, State: Initialized, Role: FOLLOWER
I20260620 21:57:55.075253 31073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.075820 31148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:55.076668 31073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:55.077045 31148 log.cc:824] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:55.080487 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: No bootstrap required, opened a new log
I20260620 21:57:55.082538 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.072884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:36691 (local address 127.0.0.1:36412)
0620 21:57:55.072979 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:55.072992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:55.073061 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:55.073326 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:55.073329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:55.073487 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:55.073638 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.073645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.074591 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.074594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.075350 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:55.075360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.075973 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.075976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:55.076040 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.076523 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:55.076536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:55.078073 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:55.080345 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.080349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.080351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.080615 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.080918 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:55.080922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:55.080924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:55.081006 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:55.081772 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:55.081775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:55.082104 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 21:57:55.082198 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:57:55.082337 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 21:57:55.083078 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.073022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.17.67.252:36691)
0620 21:57:55.073120 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.073124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.073138 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:57:55.073172 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.073175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.073213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.073298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.073754 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.074475 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.075472 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.075791 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.075881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.078192 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.078196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.078199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.078226 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.080225 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.080733 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.080736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.080738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.080820 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.081108 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.081110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.081112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.081660 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.082718 (+ 1058us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.082808 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.082931 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 21:57:55.085023 31244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:57:55.085101 31244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:57:55.085319 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.085417 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.082034 31005 master_runner.cc:186] Error getting master registration for 127.17.67.253:36257: OK, instance_id { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" instance_seqno: 1781992674798763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:57:55.085860 31005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:57:55.086476 31224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: Bootstrap starting.
I20260620 21:57:55.088079 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.062101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.17.67.254:38711)
0620 21:57:55.062310 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.062314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.068394 (+ 6080us) server_negotiation.cc:407] Connection header received
0620 21:57:55.068585 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.068588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.068632 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.068696 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.075107 (+ 6411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.075902 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.077443 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.077834 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.077917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.080990 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.080997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.081002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.081028 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.086396 (+ 5368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.086856 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.086862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.086864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.086947 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.087228 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.087233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.087234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.087412 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.087601 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.087820 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.087941 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 21:57:55.091001 31148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.092717 31224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: Neither blocks nor log segments found. Creating new log.
Jun 20 21:57:55 dist-test-slave-cx8d krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
I20260620 21:57:55.093667 31148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:55.093758 31148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c9453786a0418cbb1ecf53c82cc83d, State: Initialized, Role: FOLLOWER
I20260620 21:57:55.093890 31224 log.cc:824] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:55.094321 31148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.095808 31148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:55.096390 31224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: No bootstrap required, opened a new log
I20260620 21:57:55.097998 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.089126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.17.67.253:36257)
0620 21:57:55.089234 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.089238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.089299 (+ 61us) server_negotiation.cc:407] Connection header received
0620 21:57:55.089430 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.089433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.089483 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.089567 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.090176 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.090695 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.091452 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.091690 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.091800 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.094207 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.094211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.094213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.094239 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.095897 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.096386 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.096390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.096392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.096471 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.096826 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.096831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.096833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.097048 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.097233 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.097669 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.097790 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:57:55.101495 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:57:55.101496 31247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.101650 31247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:57:55.102591 31258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:57:55.102659 31258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:57:55.104628 31080 master_runner.cc:186] Error getting master registration for 127.17.67.252:36691: OK, instance_id { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" instance_seqno: 1781992675031545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:57:55.110387 31080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:57:55.111171 31224 raft_consensus.cc:348] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.111492 31224 raft_consensus.cc:374] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:55.111599 31224 raft_consensus.cc:729] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30f6f0683ad4bf79ed0afc917b72d70, State: Initialized, Role: FOLLOWER
I20260620 21:57:55.112190 31224 consensus_queue.cc:260] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.113673 31224 sys_catalog.cc:564] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:55.120203 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.109945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.17.67.254:38711)
0620 21:57:55.110069 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.110074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.110837 (+ 763us) server_negotiation.cc:407] Connection header received
0620 21:57:55.111024 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.111027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.111077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.111128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.112119 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.112914 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.114143 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.114478 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.114552 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.116538 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.116544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.116551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.116585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.118319 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.118813 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.118816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.118818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.118897 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.119182 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.119185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.119187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.119396 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.119521 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.119884 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.119997 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:57:55.121907 31155 master_runner.cc:263] No leader master found from master b30f6f0683ad4bf79ed0afc917b72d70
I20260620 21:57:55.121989 31155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 21:57:55.124974 31278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:57:55.125461 31278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:57:55.125304 31267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.125687 31267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.131261 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.110161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.17.67.252:36691)
0620 21:57:55.110288 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.110293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.121711 (+ 11418us) server_negotiation.cc:407] Connection header received
0620 21:57:55.121874 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.121877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.121923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.121972 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.122792 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.123561 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.124664 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.125000 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.125068 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.126776 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.126780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.126783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.126808 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.128447 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.129253 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.129257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.129259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.129347 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.129727 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.129732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.129737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.129963 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.130075 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.130720 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.131084 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:57:55.133632 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:55.110147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.17.67.253:36257)
0620 21:57:55.110268 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:55.110274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:55.115468 (+ 5194us) server_negotiation.cc:407] Connection header received
0620 21:57:55.115657 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:55.115660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:55.115719 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:55.115776 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:55.125659 (+ 9883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.126533 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.127830 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:55.128198 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:55.128288 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:55.129945 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:55.129949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:55.129952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:55.129975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:55.131624 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.132071 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.132075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.132077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.132171 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:55.132436 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:55.132439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:55.132441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:55.132633 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:55.132747 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:55.133346 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 21:57:55.133456 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:57:55.257333 31247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:57:55.257524 31247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.258647 31247 leader_election.cc:288] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04e47ebb840e4562812f19dc2f93c49d (127.17.67.254:38711), b30f6f0683ad4bf79ed0afc917b72d70 (127.17.67.252:36691)
I20260620 21:57:55.260002 31198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36c9453786a0418cbb1ecf53c82cc83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" is_pre_election: true
I20260620 21:57:55.260367 31198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c9453786a0418cbb1ecf53c82cc83d in term 0.
I20260620 21:57:55.260737 31047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36c9453786a0418cbb1ecf53c82cc83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e47ebb840e4562812f19dc2f93c49d" is_pre_election: true
I20260620 21:57:55.261022 31047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c9453786a0418cbb1ecf53c82cc83d in term 0.
I20260620 21:57:55.261432 31098 leader_election.cc:302] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e47ebb840e4562812f19dc2f93c49d, 36c9453786a0418cbb1ecf53c82cc83d; no voters:
I20260620 21:57:55.262065 31247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:57:55.262161 31247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:57:55.262208 31247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:55.263481 31247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.263813 31247 leader_election.cc:288] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [CANDIDATE]: Term 1 election: Requested vote from peers 04e47ebb840e4562812f19dc2f93c49d (127.17.67.254:38711), b30f6f0683ad4bf79ed0afc917b72d70 (127.17.67.252:36691)
I20260620 21:57:55.264449 31198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36c9453786a0418cbb1ecf53c82cc83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30f6f0683ad4bf79ed0afc917b72d70"
I20260620 21:57:55.264573 31198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:55.265708 31198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c9453786a0418cbb1ecf53c82cc83d in term 1.
I20260620 21:57:55.266032 31099 leader_election.cc:302] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c9453786a0418cbb1ecf53c82cc83d, b30f6f0683ad4bf79ed0afc917b72d70; no voters:
I20260620 21:57:55.266276 31047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36c9453786a0418cbb1ecf53c82cc83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04e47ebb840e4562812f19dc2f93c49d"
I20260620 21:57:55.266397 31047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:55.267593 31047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c9453786a0418cbb1ecf53c82cc83d in term 1.
I20260620 21:57:55.268059 31247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:55.268249 31247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [term 1 LEADER]: Becoming Leader. State: Replica: 36c9453786a0418cbb1ecf53c82cc83d, State: Running, Role: LEADER
I20260620 21:57:55.268627 31247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } }
I20260620 21:57:55.270530 31247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c9453786a0418cbb1ecf53c82cc83d. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.270637 31247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:55.273734 31282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:55.274735 31282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 21:57:55.277833 31177 tablet.cc:2369] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:57:55.279592 31198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [term 1 FOLLOWER]: Refusing update from remote peer 36c9453786a0418cbb1ecf53c82cc83d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:55.279629 31047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [term 1 FOLLOWER]: Refusing update from remote peer 36c9453786a0418cbb1ecf53c82cc83d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:57:55.280212 31247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:55.280619 31247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:57:55.283924 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c9453786a0418cbb1ecf53c82cc83d. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.284077 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.284395 31267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c9453786a0418cbb1ecf53c82cc83d. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.284523 31267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.287031 31247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.287164 31247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:55.287479 31247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.287582 31247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:55.288611 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.288733 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.288991 31283 mvcc.cc:205] Tried to move back new op lower bound from 7299041997937664000 to 7299041997904007168. Current Snapshot: MvccSnapshot[applied={T|T < 7299041997937664000}]
I20260620 21:57:55.290721 31284 mvcc.cc:205] Tried to move back new op lower bound from 7299041997937664000 to 7299041997904007168. Current Snapshot: MvccSnapshot[applied={T|T < 7299041997937664000}]
I20260620 21:57:55.291635 31267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36c9453786a0418cbb1ecf53c82cc83d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04e47ebb840e4562812f19dc2f93c49d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38711 } } peers { permanent_uuid: "36c9453786a0418cbb1ecf53c82cc83d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 36257 } } peers { permanent_uuid: "b30f6f0683ad4bf79ed0afc917b72d70" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 36691 } } }
I20260620 21:57:55.291723 31267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:57:55.292095 31285 mvcc.cc:205] Tried to move back new op lower bound from 7299041997937664000 to 7299041997904007168. Current Snapshot: MvccSnapshot[applied={T|T < 7299041997937664000}]
I20260620 21:57:55.292512 31282 catalog_manager.cc:1347] Generated new cluster ID: bca5054c884b451baca6d6e6b8e3bcbb
I20260620 21:57:55.292608 31282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:55.313488 31282 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:57:55.314535 31282 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:55.321033 31282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c9453786a0418cbb1ecf53c82cc83d: Generated new TSK 0
I20260620 21:57:55.321924 31282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:57:56.086396 31244 catalog_manager.cc:1259] Loaded cluster ID: bca5054c884b451baca6d6e6b8e3bcbb
I20260620 21:57:56.086517 31244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: loading cluster ID for follower catalog manager: success
I20260620 21:57:56.088153 31244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: acquiring CA information for follower catalog manager: success
I20260620 21:57:56.089322 31244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04e47ebb840e4562812f19dc2f93c49d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:57:56.126837 31278 catalog_manager.cc:1259] Loaded cluster ID: bca5054c884b451baca6d6e6b8e3bcbb
I20260620 21:57:56.126924 31278 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: loading cluster ID for follower catalog manager: success
I20260620 21:57:56.128604 31278 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: acquiring CA information for follower catalog manager: success
I20260620 21:57:56.129802 31278 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b30f6f0683ad4bf79ed0afc917b72d70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:57:56.381224 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31005
I20260620 21:57:56.387563 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31080
I20260620 21:57:56.393710 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31155
2026-06-20T21:57:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3164 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:57:56 dist-test-slave-cx8d krb5kdc[31297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:57:56 dist-test-slave-cx8d krb5kdc[31297](info): set up 2 sockets
Jun 20 21:57:56 dist-test-slave-cx8d krb5kdc[31297](info): commencing operation
krb5kdc: starting...
W20260620 21:57:57.438066 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:57:57.562398 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35101
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:57.672829 31310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:57.673126 31310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:57.673229 31310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:57.677312 31310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:57.677383 31310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:57.677419 31310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:57.677476 31310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:57.677532 31310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:57.682747 31310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:57.683801 31310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:57.684808 31310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:57.693145 31316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:57.693138 31315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:57.693138 31318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:57.693291 31310 server_base.cc:1034] running on GCE node
I20260620 21:57:57.693987 31310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:57.694983 31310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:57.696137 31310 hybrid_clock.cc:648] HybridClock initialized: now 1781992677696109 us; error 42 us; skew 500 ppm
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992677, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:57.698930 31310 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:57.699759 31310 webserver.cc:460] Webserver started at http://127.17.67.254:45111/ using document root <none> and password file <none>
I20260620 21:57:57.700263 31310 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:57.700318 31310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:57.700477 31310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:57.702455 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "142c2b69e6b34c97a9e25bfd6164da0d"
format_stamp: "Formatted at 2026-06-20 21:57:57 on dist-test-slave-cx8d"
server_key: "9e774a44c3d0cab9c8f7a72a6294b7db"
server_key_iv: "b49602ac39147babfbde82ff6f657f43"
server_key_version: "encryptionkey@0"
I20260620 21:57:57.702947 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "142c2b69e6b34c97a9e25bfd6164da0d"
format_stamp: "Formatted at 2026-06-20 21:57:57 on dist-test-slave-cx8d"
server_key: "9e774a44c3d0cab9c8f7a72a6294b7db"
server_key_iv: "b49602ac39147babfbde82ff6f657f43"
server_key_version: "encryptionkey@0"
I20260620 21:57:57.706527 31310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:57:57.708737 31325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:57.709975 31310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:57:57.710078 31310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "142c2b69e6b34c97a9e25bfd6164da0d"
format_stamp: "Formatted at 2026-06-20 21:57:57 on dist-test-slave-cx8d"
server_key: "9e774a44c3d0cab9c8f7a72a6294b7db"
server_key_iv: "b49602ac39147babfbde82ff6f657f43"
server_key_version: "encryptionkey@0"
I20260620 21:57:57.710204 31310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:57.722429 31310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:57.725770 31310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:57.725965 31310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:57.733119 31310 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:35101
I20260620 21:57:57.733218 31377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:35101 every 8 connection(s)
I20260620 21:57:57.734174 31310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:57.736948 31378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:57.738166 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31310
I20260620 21:57:57.738307 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:57:57.738633 17679 external_mini_cluster.cc:1428] Setting key b45d606ee9fae093e2dd8d0048be9df1
I20260620 21:57:57.743224 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d: Bootstrap starting.
I20260620 21:57:57.745419 31378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:57.746321 31378 log.cc:824] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:57.748204 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d: No bootstrap required, opened a new log
I20260620 21:57:57.751799 31378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } }
I20260620 21:57:57.752085 31378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:57.752171 31378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142c2b69e6b34c97a9e25bfd6164da0d, State: Initialized, Role: FOLLOWER
I20260620 21:57:57.752622 31378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } }
I20260620 21:57:57.752740 31378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:57:57.752820 31378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:57:57.752929 31378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:57:57.753021 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:57.740138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.17.67.254:35101)
0620 21:57:57.740594 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:57.740609 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:57.740643 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:57:57.741319 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:57.741336 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:57.741739 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:57.742084 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:57.742885 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:57.743916 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:57.744778 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:57.745250 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:57.745427 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:57.747983 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:57.747994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:57.748008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:57.748034 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:57.749751 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:57.750175 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:57.750181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:57.750196 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:57.750302 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:57.750602 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:57.750607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:57.750608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:57.751038 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:57.751375 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:57.751670 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 21:57:57.751998 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260620 21:57:57.753979 31378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } }
I20260620 21:57:57.754324 31378 leader_election.cc:302] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142c2b69e6b34c97a9e25bfd6164da0d; no voters:
I20260620 21:57:57.754664 31378 leader_election.cc:288] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:57:57.754743 31383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:57.754932 31383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [term 1 LEADER]: Becoming Leader. State: Replica: 142c2b69e6b34c97a9e25bfd6164da0d, State: Running, Role: LEADER
I20260620 21:57:57.755218 31383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } }
I20260620 21:57:57.755923 31378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:57.756349 31383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142c2b69e6b34c97a9e25bfd6164da0d. Latest consensus state: current_term: 1 leader_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } } }
I20260620 21:57:57.756453 31383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:57.756495 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:57:57.756825 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142c2b69e6b34c97a9e25bfd6164da0d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 35101 } } }
I20260620 21:57:57.756945 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:57.756834 31387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:57.760535 31387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:57:57.767272 31387 catalog_manager.cc:1347] Generated new cluster ID: c9f844926d5241b0b57f7a100810ecd1
I20260620 21:57:57.767371 31387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:57.784318 31387 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:57:57.785323 31387 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:57.796615 31387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142c2b69e6b34c97a9e25bfd6164da0d: Generated new TSK 0
I20260620 21:57:57.797360 31387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:57:57.813210 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31310
2026-06-20T21:57:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31405](info): set up 2 sockets
Jun 20 21:57:57 dist-test-slave-cx8d krb5kdc[31405](info): commencing operation
krb5kdc: starting...
W20260620 21:57:58.861877 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:57:58 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:57:58.982479 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:59.096026 31418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:59.096287 31418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:59.096328 31418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:59.100323 31418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:57:59.100381 31418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:59.100401 31418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:59.100417 31418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:57:59.100432 31418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:57:59.105630 31418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:59.106593 31418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:59.107553 31418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:59.115934 31426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:59.116036 31423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:59.115934 31424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.116325 31418 server_base.cc:1034] running on GCE node
I20260620 21:57:59.116845 31418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:59.118111 31418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:59.119302 31418 hybrid_clock.cc:648] HybridClock initialized: now 1781992679119275 us; error 42 us; skew 500 ppm
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:59.125252 31418 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:57:59.126192 31418 webserver.cc:460] Webserver started at http://127.17.67.254:34589/ using document root <none> and password file <none>
I20260620 21:57:59.126746 31418 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:59.126837 31418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:59.127074 31418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:59.129145 31418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "9ec51dfc56ca46d8913cc86e61ec49b7"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "2cedafb23e74df210f459efa74213156"
server_key_iv: "8c1ea5010b765e4789ef1ca352f7eaa1"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.129786 31418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "9ec51dfc56ca46d8913cc86e61ec49b7"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "2cedafb23e74df210f459efa74213156"
server_key_iv: "8c1ea5010b765e4789ef1ca352f7eaa1"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.133427 31418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:57:59.135870 31433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.137063 31418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:57:59.137202 31418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "9ec51dfc56ca46d8913cc86e61ec49b7"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "2cedafb23e74df210f459efa74213156"
server_key_iv: "8c1ea5010b765e4789ef1ca352f7eaa1"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.137313 31418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:59.158635 31418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:59.161844 31418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:59.162052 31418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:59.168815 31418 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:46181
I20260620 21:57:59.169126 31485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:46181 every 8 connection(s)
I20260620 21:57:59.169823 31418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:57:59.172402 31486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:59.177707 31486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap starting.
I20260620 21:57:59.178977 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31418
I20260620 21:57:59.179128 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:57:59.179446 17679 external_mini_cluster.cc:1428] Setting key 06c78598145ef50b256fb4d05e0b1b7c
I20260620 21:57:59.180282 31486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:59.180997 31486 log.cc:824] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:59.182981 31486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: No bootstrap required, opened a new log
I20260620 21:57:59.186921 31486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:57:59.187206 31486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:59.187268 31486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Initialized, Role: FOLLOWER
I20260620 21:57:59.187825 31486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:57:59.188007 31486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:57:59.188088 31486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:57:59.188196 31486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:59.189095 31486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:57:59.189455 31486 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec51dfc56ca46d8913cc86e61ec49b7; no voters:
I20260620 21:57:59.189782 31486 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:57:59.189888 31491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:57:59.190191 31491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Running, Role: LEADER
I20260620 21:57:59.190548 31491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:57:59.191094 31486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:57:59.192708 31493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec51dfc56ca46d8913cc86e61ec49b7. Latest consensus state: current_term: 1 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:57:59.192834 31493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
I20260620 21:57:59.192903 31492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:57:59.193001 31492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
W20260620 21:57:59.195973 31506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:57:59.196046 31506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:57:59.196105 31505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:57:59.196139 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.180612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.17.67.254:46181)
0620 21:57:59.181095 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.181134 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.181179 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:57:59.182125 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.182146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.182496 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.182788 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:59.183663 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.184764 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.186091 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.186569 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.186787 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.189818 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:59.189826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:59.189846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:59.189883 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:59.192311 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.192827 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.192833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.192841 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.192997 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.193430 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.193435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.193437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.194022 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:59.194242 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.194939 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.195232 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":160,"threads_started":1}
I20260620 21:57:59.196919 31505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:57:59.202425 31505 catalog_manager.cc:1347] Generated new cluster ID: 8b86a3577f4b4bd18553f5719f07bace
I20260620 21:57:59.202498 31505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:57:59.228153 31505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:57:59.229163 31505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:57:59.239697 31505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Generated new TSK 0
I20260620 21:57:59.240374 31505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:57:59.268213 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:59.377074 31512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:59.377331 31512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:59.377375 31512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:59.381490 31512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:59.381580 31512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:59.381686 31512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:57:59.386942 31512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:59.388002 31512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:59.389038 31512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:59.398792 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:59.398797 31518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.398963 31512 server_base.cc:1034] running on GCE node
W20260620 21:57:59.398823 31517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.399557 31512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:59.400244 31512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:59.401429 31512 hybrid_clock.cc:648] HybridClock initialized: now 1781992679401415 us; error 29 us; skew 500 ppm
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:59.404302 31512 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:57:59.405165 31512 webserver.cc:460] Webserver started at http://127.17.67.193:41915/ using document root <none> and password file <none>
I20260620 21:57:59.405777 31512 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:59.405839 31512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:59.406080 31512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:59.408113 31512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "ce803d93230e4feda73d3213bd06de74"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "7e00e19784245446972583ce3639038f"
server_key_iv: "7a112f9398d9b884a392c9b922a53c9a"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.408680 31512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "ce803d93230e4feda73d3213bd06de74"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "7e00e19784245446972583ce3639038f"
server_key_iv: "7a112f9398d9b884a392c9b922a53c9a"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.412456 31512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:57:59.414930 31527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.416139 31512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:57:59.416286 31512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "ce803d93230e4feda73d3213bd06de74"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "7e00e19784245446972583ce3639038f"
server_key_iv: "7a112f9398d9b884a392c9b922a53c9a"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.416404 31512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:59.431418 31512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:59.434688 31512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:59.434900 31512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:59.435555 31512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:59.436388 31512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:59.436460 31512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.436530 31512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:59.436578 31512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.445647 31512 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:41543
I20260620 21:57:59.445672 31640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:41543 every 8 connection(s)
I20260620 21:57:59.446527 31512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:57:59.454486 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31512
I20260620 21:57:59.454646 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:57:59.454948 17679 external_mini_cluster.cc:1428] Setting key 542acbbdae0e7e6cbd0fa9e41c1329a5
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:59.461203 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.448323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:42565 (local address 127.17.67.254:46181)
0620 21:57:59.448461 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.448465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.449402 (+ 937us) server_negotiation.cc:407] Connection header received
0620 21:57:59.450838 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.450841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.450910 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.450994 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:59.452251 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.452800 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.454071 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.454317 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.454431 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.457236 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:59.457243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:59.457249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:59.457282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:59.458826 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.459338 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.459342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.459346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.459460 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.459817 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.459820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.459822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.460051 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:59.460152 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.460845 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.460988 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:57:59.462009 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.448727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:42565)
0620 21:57:59.449233 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.449277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.450611 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.451232 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.451239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.451605 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:59.452094 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.452107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.452929 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.452933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.453934 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.453943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.454471 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.454475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.454707 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.455382 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:59.455399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:59.457069 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:59.458950 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.458967 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.458981 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.459233 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.459563 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.459567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.459569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.459714 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.460190 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:59.460201 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:59.460542 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.460759 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.461071 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260620 21:57:59.463389 31641 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:57:59.463665 31641 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:59.464282 31641 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:57:59.465910 31450 ts_manager.cc:194] Registered new tserver with Master: ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543)
I20260620 21:57:59.466990 31450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:42565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:57:59.482770 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:59.592970 31646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:59.593240 31646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:59.593286 31646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:59.597265 31646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:59.597316 31646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:59.597374 31646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:57:59.602432 31646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:59.603391 31646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:59.604359 31646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:59.613832 31652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:59.613819 31651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.614158 31646 server_base.cc:1034] running on GCE node
W20260620 21:57:59.613867 31654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.614651 31646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:59.615217 31646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:59.616374 31646 hybrid_clock.cc:648] HybridClock initialized: now 1781992679616334 us; error 55 us; skew 500 ppm
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:59.619155 31646 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:57:59.620035 31646 webserver.cc:460] Webserver started at http://127.17.67.194:44483/ using document root <none> and password file <none>
I20260620 21:57:59.620627 31646 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:59.620689 31646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:59.620898 31646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:59.622941 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "f643911191884100a8a8d92e0bd01a0f"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "fb71b95d00cfa38065f49555b52d57fc"
server_key_iv: "2819204de17cfcf875d165cfcbc0228d"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.623502 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "f643911191884100a8a8d92e0bd01a0f"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "fb71b95d00cfa38065f49555b52d57fc"
server_key_iv: "2819204de17cfcf875d165cfcbc0228d"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.627310 31646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 21:57:59.629781 31661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.630995 31646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:57:59.631085 31646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "f643911191884100a8a8d92e0bd01a0f"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "fb71b95d00cfa38065f49555b52d57fc"
server_key_iv: "2819204de17cfcf875d165cfcbc0228d"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.631234 31646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:59.647466 31646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:59.650282 31646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:59.650437 31646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:59.650936 31646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:59.651667 31646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:59.651714 31646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.651751 31646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:59.651767 31646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.660924 31646 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40327
I20260620 21:57:59.660948 31774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40327 every 8 connection(s)
I20260620 21:57:59.661873 31646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:57:59.669304 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31646
I20260620 21:57:59.669476 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:57:59.669826 17679 external_mini_cluster.cc:1428] Setting key d15b93772ae589aa4fdebf7f9f077dd6
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:59.676455 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.663835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:46313 (local address 127.17.67.254:46181)
0620 21:57:59.664012 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.664017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.664906 (+ 889us) server_negotiation.cc:407] Connection header received
0620 21:57:59.666280 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.666283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.666330 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.666403 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:59.667587 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.668090 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.669407 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.669734 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.669815 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.672652 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:59.672658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:59.672665 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:59.672697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:59.674191 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.674816 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.674819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.674820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.674892 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.675291 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.675293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.675294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.675514 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:59.675616 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.676238 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.676333 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:57:59.677807 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.664275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:46313)
0620 21:57:59.664745 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.664788 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.666073 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.666573 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.666580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.666928 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:59.667465 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.667477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.668210 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.668213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.669262 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.669273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.669887 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.669893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.670104 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.670836 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:59.670855 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:59.672462 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:59.674307 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.674316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.674334 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.674713 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.674986 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.674992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.674997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.675198 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.675635 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:59.675644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:59.675982 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.676207 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.676606 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260620 21:57:59.679044 31775 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:57:59.679299 31775 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:57:59.679782 31775 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
I20260620 21:57:59.680912 31450 ts_manager.cc:194] Registered new tserver with Master: f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327)
I20260620 21:57:59.681448 31450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:46313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:57:59.696599 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:57:59.807391 31780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:57:59.807677 31780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:57:59.807721 31780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:57:59.811878 31780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:57:59.811936 31780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:57:59.812032 31780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:57:59.817313 31780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:57:59.818405 31780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:57:59.819435 31780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:57:59.829332 31786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:57:59.829319 31788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.829457 31780 server_base.cc:1034] running on GCE node
W20260620 21:57:59.829381 31785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:57:59.829988 31780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:57:59.830678 31780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:57:59.831863 31780 hybrid_clock.cc:648] HybridClock initialized: now 1781992679831847 us; error 26 us; skew 500 ppm
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:57:59.834894 31780 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:57:59.835913 31780 webserver.cc:460] Webserver started at http://127.17.67.195:38091/ using document root <none> and password file <none>
I20260620 21:57:59.836544 31780 fs_manager.cc:359] Metadata directory not provided
I20260620 21:57:59.836632 31780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:57:59.836844 31780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:57:59.838948 31780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "579c3a7f2fb71af75d639a62a6cff06b"
server_key_iv: "e5597307a3162c9f2e189110a0ebe98e"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.839537 31780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "579c3a7f2fb71af75d639a62a6cff06b"
server_key_iv: "e5597307a3162c9f2e189110a0ebe98e"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.843811 31780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:57:59.846457 31795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.847719 31780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:57:59.847867 31780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "579c3a7f2fb71af75d639a62a6cff06b"
server_key_iv: "e5597307a3162c9f2e189110a0ebe98e"
server_key_version: "encryptionkey@0"
I20260620 21:57:59.848034 31780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:57:59.866731 31780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:57:59.871253 31780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:57:59.871495 31780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:57:59.872270 31780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:57:59.873359 31780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:57:59.873409 31780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.873450 31780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:57:59.873505 31780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:57:59.885107 31780 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:46315
I20260620 21:57:59.885123 31908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:46315 every 8 connection(s)
I20260620 21:57:59.886135 31780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:57:59.893343 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31780
I20260620 21:57:59.893467 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:57:59.893765 17679 external_mini_cluster.cc:1428] Setting key 7db61055059d30dd7749b0488ce5da41
Jun 20 21:57:59 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992679, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:57:59.901842 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.888004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:56617 (local address 127.17.67.254:46181)
0620 21:57:59.888177 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.888181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.888940 (+ 759us) server_negotiation.cc:407] Connection header received
0620 21:57:59.890474 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.890478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.890540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.890671 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:59.892047 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.892592 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.894162 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.894483 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.894624 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.897953 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:59.897957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:59.897959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:59.897986 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:59.899506 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.900124 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.900127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.900129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.900214 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.900592 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.900595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.900596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.900810 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:59.900894 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.901523 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.901716 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 21:57:59.902915 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.888320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:56617)
0620 21:57:59.888767 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.888812 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.890251 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.890846 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.890853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.891274 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:57:59.891879 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.891895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.892721 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.892731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.894024 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.894034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.894735 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.894740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.895056 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.895870 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:57:59.895907 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:57:59.897678 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:57:59.899628 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.899633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.899647 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.899982 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.900312 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:57:59.900314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:57:59.900316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:57:59.900486 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:57:59.900945 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:57:59.900951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:57:59.901222 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.901429 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.901962 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260620 21:57:59.904256 31909 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:57:59.904563 31909 heartbeater.cc:464] Registering TS with master...
I20260620 21:57:59.905102 31909 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
I20260620 21:57:59.906394 31450 ts_manager.cc:194] Registered new tserver with Master: 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315)
I20260620 21:57:59.907083 31450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:56617
I20260620 21:57:59.908026 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:57:59.918468 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.910394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.17.67.254:46181)
0620 21:57:59.910569 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.910574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.910752 (+ 178us) server_negotiation.cc:407] Connection header received
0620 21:57:59.910863 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.910865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.910904 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.910969 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:57:59.911665 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.912171 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.913182 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.913393 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.913468 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.915191 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:57:59.915194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:57:59.915197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:57:59.915219 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:57:59.916705 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.917145 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.917147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.917149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.917221 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:57:59.917478 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:57:59.917480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:57:59.917481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:57:59.917770 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:57:59.917875 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.918165 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.918261 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:57:59.921830 31450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:57:59.924809 31450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:57:59.939152 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.933431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.0.0.1:41924)
0620 21:57:59.934151 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.934168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.934252 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.934794 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.934801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.934876 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:59.935076 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.935083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936263 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.936266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.937732 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.937741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.938615 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.938619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.938731 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.938778 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.938870 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.938988 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260620 21:57:59.939942 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.933024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:46315 (local address 127.0.0.1:60204)
0620 21:57:59.933411 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.933464 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.933614 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.935062 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.935065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.935083 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:59.935325 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.935336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936988 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.936991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.938468 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.938480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.939581 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.939584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.939669 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.939690 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.939741 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.939812 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":81,"thread_start_us":95,"threads_started":1}
I20260620 21:57:59.940011 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.933811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.17.67.194:40327)
0620 21:57:59.934252 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.934263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.934288 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:57:59.934371 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.934376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.934626 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.934783 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:59.935213 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936098 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.937849 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.938451 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.939551 (+ 1100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.939725 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.939777 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.939873 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":143,"threads_started":1}
I20260620 21:57:59.940125 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.933854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.0.0.1:52100)
0620 21:57:59.934370 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 21:57:59.934386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:57:59.934476 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:57:59.934976 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:57:59.934979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:57:59.934997 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:57:59.935268 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.935275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936898 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.936905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.938468 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:57:59.938479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.939623 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.939630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:57:59.939753 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.939779 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:57:59.939861 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 21:57:59.939957 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":293,"thread_start_us":94,"threads_started":1}
I20260620 21:57:59.940651 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.934089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.17.67.193:41543)
0620 21:57:59.934558 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.934575 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.934598 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:57:59.934668 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.934675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.934856 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.934997 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:59.935387 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936724 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.938634 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.939289 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.939405 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.939556 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.940389 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.940506 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":61,"threads_started":1}
I20260620 21:57:59.940680 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:57:59.933175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.17.67.195:46315)
0620 21:57:59.934552 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0620 21:57:59.934566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:57:59.934593 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:57:59.934663 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:57:59.934667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:57:59.934896 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:57:59.935478 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:57:59.935664 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.936847 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.938602 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:57:59.939436 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:57:59.940240 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:57:59.940373 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:57:59.940420 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 21:57:59.940546 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":166,"threads_started":1}
I20260620 21:57:59.943193 31843 tablet_service.cc:1467] Processing CreateTablet for tablet 5711bf89a39c45e88428efb979a28900 (DEFAULT_TABLE table=test-table [id=3a5d3011faca468ab1c647fcb7591376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:59.943387 31709 tablet_service.cc:1467] Processing CreateTablet for tablet 5711bf89a39c45e88428efb979a28900 (DEFAULT_TABLE table=test-table [id=3a5d3011faca468ab1c647fcb7591376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:59.943389 31575 tablet_service.cc:1467] Processing CreateTablet for tablet 5711bf89a39c45e88428efb979a28900 (DEFAULT_TABLE table=test-table [id=3a5d3011faca468ab1c647fcb7591376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:57:59.944272 31843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5711bf89a39c45e88428efb979a28900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:59.944480 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5711bf89a39c45e88428efb979a28900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:59.944480 31709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5711bf89a39c45e88428efb979a28900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:57:59.951169 31926 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Bootstrap starting.
I20260620 21:57:59.952073 31928 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Bootstrap starting.
I20260620 21:57:59.953095 31927 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Bootstrap starting.
I20260620 21:57:59.953889 31926 tablet_bootstrap.cc:654] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:59.954717 31928 tablet_bootstrap.cc:654] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:59.954957 31927 tablet_bootstrap.cc:654] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Neither blocks nor log segments found. Creating new log.
I20260620 21:57:59.955305 31926 log.cc:824] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:59.955444 31928 log.cc:824] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:59.956287 31927 log.cc:824] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Log is configured to *not* fsync() on all Append() calls
I20260620 21:57:59.957458 31928 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: No bootstrap required, opened a new log
I20260620 21:57:59.957477 31926 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: No bootstrap required, opened a new log
I20260620 21:57:59.957695 31928 ts_tablet_manager.cc:1397] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:57:59.957701 31926 ts_tablet_manager.cc:1397] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:57:59.958197 31927 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: No bootstrap required, opened a new log
I20260620 21:57:59.958374 31927 ts_tablet_manager.cc:1397] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:57:59.960572 31928 raft_consensus.cc:348] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.960803 31928 raft_consensus.cc:374] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:59.960855 31928 raft_consensus.cc:729] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f643911191884100a8a8d92e0bd01a0f, State: Initialized, Role: FOLLOWER
I20260620 21:57:59.961293 31928 consensus_queue.cc:260] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.961948 31775 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
I20260620 21:57:59.961869 31926 raft_consensus.cc:348] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.962129 31926 raft_consensus.cc:374] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:59.962193 31926 raft_consensus.cc:729] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e7ac8b4cc34e4b8ea057cc796d0b1c, State: Initialized, Role: FOLLOWER
I20260620 21:57:59.962399 31927 raft_consensus.cc:348] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.962627 31926 consensus_queue.cc:260] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.962639 31927 raft_consensus.cc:374] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:57:59.962873 31927 raft_consensus.cc:729] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce803d93230e4feda73d3213bd06de74, State: Initialized, Role: FOLLOWER
I20260620 21:57:59.963376 31928 ts_tablet_manager.cc:1428] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260620 21:57:59.963359 31927 consensus_queue.cc:260] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:57:59.963642 31926 ts_tablet_manager.cc:1428] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:57:59.963999 31909 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
I20260620 21:57:59.964236 31927 ts_tablet_manager.cc:1428] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:57:59.964552 31641 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
I20260620 21:58:00.041950 31933 raft_consensus.cc:482] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:00.042227 31933 raft_consensus.cc:504] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:58:00.043515 31933 leader_election.cc:288] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543), f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327)
I20260620 21:58:00.047375 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.043819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:58275 (local address 127.17.67.193:41543)
0620 21:58:00.043957 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.043962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.043989 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:58:00.044134 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.044138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.044205 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.044314 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:00.044606 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.045163 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.046361 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.046782 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.046920 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.046979 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.047182 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.047258 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:58:00.047376 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.043734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.17.67.195:58275)
0620 21:58:00.043852 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.043869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.043967 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.044326 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.044329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.044349 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:00.044476 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.044482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.045285 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.045291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.046225 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.046235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.046908 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.046912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.047027 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.047057 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.047150 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.047227 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":35}
I20260620 21:58:00.047479 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.044102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:39683 (local address 127.17.67.194:40327)
0620 21:58:00.044250 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.044255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.044377 (+ 122us) server_negotiation.cc:407] Connection header received
0620 21:58:00.044531 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.044534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.044575 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.044660 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:00.044921 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.045393 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.046525 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.046910 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.047042 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.047092 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.047267 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.047333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:58:00.047536 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.043794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.17.67.195:39683)
0620 21:58:00.044277 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.044290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.044407 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.044668 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.044671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.044685 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:00.044808 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.044812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.045500 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.045507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.046399 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.046408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.047032 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.047036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.047137 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.047159 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.047253 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.047321 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":51,"thread_start_us":177,"threads_started":1}
I20260620 21:58:00.048117 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5711bf89a39c45e88428efb979a28900" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f" is_pre_election: true
I20260620 21:58:00.048151 31595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5711bf89a39c45e88428efb979a28900" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce803d93230e4feda73d3213bd06de74" is_pre_election: true
I20260620 21:58:00.048442 31595 raft_consensus.cc:2454] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 0.
I20260620 21:58:00.048434 31729 raft_consensus.cc:2454] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 0.
I20260620 21:58:00.048987 31798 leader_election.cc:302] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e7ac8b4cc34e4b8ea057cc796d0b1c, f643911191884100a8a8d92e0bd01a0f; no voters:
I20260620 21:58:00.049322 31933 raft_consensus.cc:2781] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:00.049450 31933 raft_consensus.cc:482] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:00.049510 31933 raft_consensus.cc:3037] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:00.050565 31933 raft_consensus.cc:504] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:58:00.050952 31933 leader_election.cc:288] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 election: Requested vote from peers ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543), f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327)
I20260620 21:58:00.051326 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5711bf89a39c45e88428efb979a28900" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f"
I20260620 21:58:00.051326 31595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5711bf89a39c45e88428efb979a28900" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce803d93230e4feda73d3213bd06de74"
I20260620 21:58:00.051458 31595 raft_consensus.cc:3037] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:00.051458 31729 raft_consensus.cc:3037] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:00.053074 31729 raft_consensus.cc:2454] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 1.
I20260620 21:58:00.053107 31595 raft_consensus.cc:2454] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 1.
I20260620 21:58:00.053387 31797 leader_election.cc:302] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e7ac8b4cc34e4b8ea057cc796d0b1c, ce803d93230e4feda73d3213bd06de74; no voters:
I20260620 21:58:00.053593 31933 raft_consensus.cc:2781] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:00.053825 31933 raft_consensus.cc:686] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 LEADER]: Becoming Leader. State: Replica: 25e7ac8b4cc34e4b8ea057cc796d0b1c, State: Running, Role: LEADER
I20260620 21:58:00.054175 31933 consensus_queue.cc:237] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:58:00.057677 31448 catalog_manager.cc:5495] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195). New cstate: current_term: 1 leader_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:00.095892 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.092370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.17.67.195:46315)
0620 21:58:00.092506 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.092510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.092546 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:58:00.092720 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.092723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.092780 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.092862 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:00.093117 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.093627 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.094441 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.094650 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.094726 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.094987 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:00.095415 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:00.095503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.095644 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.095691 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:58:00.101900 31729 raft_consensus.cc:1261] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Refusing update from remote peer 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:00.101900 31595 raft_consensus.cc:1261] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Refusing update from remote peer 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:00.102679 31936 consensus_queue.cc:1035] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:00.102890 31933 consensus_queue.cc:1035] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:00.119416 31448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:00.119622 31448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:00.122190 31448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Sending DeleteTablet for 3 replicas of tablet 5711bf89a39c45e88428efb979a28900
I20260620 21:58:00.123034 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31512
I20260620 21:58:00.123045 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet 5711bf89a39c45e88428efb979a28900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:00 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:52100
I20260620 21:58:00.123090 31709 tablet_service.cc:1514] Processing DeleteTablet for tablet 5711bf89a39c45e88428efb979a28900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:00 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:41924
I20260620 21:58:00.123860 31843 tablet_service.cc:1514] Processing DeleteTablet for tablet 5711bf89a39c45e88428efb979a28900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:00 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60204
I20260620 21:58:00.123931 31949 tablet_replica.cc:330] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: stopping tablet replica
I20260620 21:58:00.124279 31949 raft_consensus.cc:2229] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:00.124466 31950 tablet_replica.cc:330] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: stopping tablet replica
I20260620 21:58:00.124641 31949 raft_consensus.cc:2258] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:00.124869 31950 raft_consensus.cc:2229] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:00.125330 31950 raft_consensus.cc:2258] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:00.126361 31949 ts_tablet_manager.cc:1905] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:00.126772 31950 ts_tablet_manager.cc:1905] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:00.129673 31949 ts_tablet_manager.cc:1918] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:00.129796 31949 log.cc:1196] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/5711bf89a39c45e88428efb979a28900
I20260620 21:58:00.129956 31950 ts_tablet_manager.cc:1918] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:00.130143 31949 ts_tablet_manager.cc:1939] T 5711bf89a39c45e88428efb979a28900 P f643911191884100a8a8d92e0bd01a0f: Deleting consensus metadata
I20260620 21:58:00.130184 31950 log.cc:1196] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/5711bf89a39c45e88428efb979a28900
I20260620 21:58:00.130630 31950 ts_tablet_manager.cc:1939] T 5711bf89a39c45e88428efb979a28900 P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting consensus metadata
I20260620 21:58:00.131608 31436 catalog_manager.cc:4841] TS f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327): tablet 5711bf89a39c45e88428efb979a28900 (table test-table [id=3a5d3011faca468ab1c647fcb7591376]) successfully deleted
W20260620 21:58:00.131991 31435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:41543 (error 108)
I20260620 21:58:00.132453 31435 catalog_manager.cc:4841] TS 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315): tablet 5711bf89a39c45e88428efb979a28900 (table test-table [id=3a5d3011faca468ab1c647fcb7591376]) successfully deleted
W20260620 21:58:00.133040 31922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:58:00.132768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.0.0.1:52110)
0620 21:58:00.132900 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.132960 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:41543: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 21:58:00.133240 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31646
W20260620 21:58:00.133234 31435 catalog_manager.cc:4584] TS ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5711bf89a39c45e88428efb979a28900: Network error: Client connection negotiation failed: client connection to 127.17.67.193:41543: connect: Connection reset by peer (error 104)
I20260620 21:58:00.139780 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31780
I20260620 21:58:00.145689 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31418
I20260620 21:58:00.152616 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--webserver_interface=127.17.67.254
--webserver_port=34589
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:00.262606 31952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:00.262902 31952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:00.262976 31952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:00.267033 31952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:00.267125 31952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:00.267163 31952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:00.267222 31952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:00.267258 31952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:00.272586 31952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:00.273696 31952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:00.274763 31952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:00.287576 31958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.287545 31961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.287572 31959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:00.287957 31952 server_base.cc:1034] running on GCE node
I20260620 21:58:00.288519 31952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:00.289757 31952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:00.290917 31952 hybrid_clock.cc:648] HybridClock initialized: now 1781992680290902 us; error 27 us; skew 500 ppm
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:00.293525 31952 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:58:00.294504 31952 webserver.cc:460] Webserver started at http://127.17.67.254:34589/ using document root <none> and password file <none>
I20260620 21:58:00.295104 31952 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:00.295217 31952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:00.299078 31952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 21:58:00.301111 31968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.302331 31952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:00.302477 31952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "9ec51dfc56ca46d8913cc86e61ec49b7"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "2cedafb23e74df210f459efa74213156"
server_key_iv: "8c1ea5010b765e4789ef1ca352f7eaa1"
server_key_version: "encryptionkey@0"
I20260620 21:58:00.303092 31952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:00.322926 31952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:00.326048 31952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:00.326272 31952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:00.333827 31952 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:46181
I20260620 21:58:00.334144 32020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:46181 every 8 connection(s)
I20260620 21:58:00.335029 31952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:00.338902 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 31952
I20260620 21:58:00.339380 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:41543
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=41915
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:00.339612 32021 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:58:00.341925 32021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap starting.
I20260620 21:58:00.358817 32021 log.cc:824] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:00.367273 32021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:00.367664 32021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap complete.
I20260620 21:58:00.371774 32021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:00.372228 32021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Initialized, Role: FOLLOWER
I20260620 21:58:00.372706 32021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:00.372814 32021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:00.372902 32021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:00.373015 32021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:58:00.374959 32021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:00.375293 32021 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec51dfc56ca46d8913cc86e61ec49b7; no voters:
I20260620 21:58:00.375583 32021 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:58:00.375725 32025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:58:00.375969 32025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 LEADER]: Becoming Leader. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Running, Role: LEADER
I20260620 21:58:00.376282 32025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:00.376784 32021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:00.377607 32026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:58:00.377756 32026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:00.378087 32027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec51dfc56ca46d8913cc86e61ec49b7. Latest consensus state: current_term: 2 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:58:00.378170 32027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:00.378544 32034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:00.383258 32034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:00.384075 32034 catalog_manager.cc:1259] Loaded cluster ID: 8b86a3577f4b4bd18553f5719f07bace
I20260620 21:58:00.384148 32034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:00.385699 32034 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 21:58:00.390123 32034 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 21:58:00.395540 32034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:00.396266 32034 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Loaded TSK: 0
I20260620 21:58:00.397435 32034 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:58:00.456609 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:00.456904 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:00.456981 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:00.461098 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:00.461197 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:00.461290 32023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:58:00.466634 32023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:41543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=41915
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:00.467655 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:00.468676 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:00.479872 32049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.479920 32048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.479857 32051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:00.479970 32023 server_base.cc:1034] running on GCE node
I20260620 21:58:00.480661 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:00.481449 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:00.482684 32023 hybrid_clock.cc:648] HybridClock initialized: now 1781992680482651 us; error 59 us; skew 500 ppm
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:00.485695 32023 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:58:00.486666 32023 webserver.cc:460] Webserver started at http://127.17.67.193:41915/ using document root <none> and password file <none>
I20260620 21:58:00.487254 32023 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:00.487329 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:00.491443 32023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:58:00.493752 32058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.495273 32023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:58:00.495522 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "ce803d93230e4feda73d3213bd06de74"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "7e00e19784245446972583ce3639038f"
server_key_iv: "7a112f9398d9b884a392c9b922a53c9a"
server_key_version: "encryptionkey@0"
I20260620 21:58:00.496150 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:00.519749 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:00.523281 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:00.523514 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:00.524183 32023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:00.525378 32065 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:58:00.529486 32023 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:58:00.529600 32023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 21:58:00.529695 32023 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:58:00.531472 32023 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:58:00.531551 32023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:58:00.531818 32065 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Bootstrap starting.
I20260620 21:58:00.542551 32023 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:41543
I20260620 21:58:00.542582 32172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:41543 every 8 connection(s)
I20260620 21:58:00.543498 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:00.546501 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32023
I20260620 21:58:00.546934 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:40327
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=44483
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:00.551064 32065 log.cc:824] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:00.558485 32065 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:00.558889 32065 tablet_bootstrap.cc:492] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Bootstrap complete.
I20260620 21:58:00.560384 32065 ts_tablet_manager.cc:1397] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.008s
I20260620 21:58:00.563205 32065 raft_consensus.cc:348] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:58:00.563742 32065 raft_consensus.cc:729] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce803d93230e4feda73d3213bd06de74, State: Initialized, Role: FOLLOWER
I20260620 21:58:00.563902 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.545501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:57087)
0620 21:58:00.546196 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.546252 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.547471 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.549162 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.549173 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.549627 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:00.550097 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.550111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.551949 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.551954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.553555 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.553588 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.554589 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.554596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.554843 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.555582 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:00.555621 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:00.557475 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:00.560286 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.560293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.560308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.560706 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.561116 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.561120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.561122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.561300 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.562076 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:00.562082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:00.562365 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.562555 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.562882 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":155,"threads_started":1}
I20260620 21:58:00.564301 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.545528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57087 (local address 127.17.67.254:46181)
0620 21:58:00.545970 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.545991 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.547802 (+ 1811us) server_negotiation.cc:407] Connection header received
0620 21:58:00.548123 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.548139 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.548620 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.549061 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:00.550281 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.551763 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.553734 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.554423 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.554590 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.557733 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:00.557740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:00.557755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:00.557789 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:00.560147 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.560851 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.560857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.560862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.561006 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.561419 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.561423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.561424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.561898 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:00.562031 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.562717 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.563016 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260620 21:58:00.564268 32065 consensus_queue.cc:260] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } } peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } }
I20260620 21:58:00.565282 32065 ts_tablet_manager.cc:1428] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 21:58:00.566833 32173 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:58:00.567075 32173 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:00.567534 32173 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
I20260620 21:58:00.569993 31984 ts_manager.cc:194] Registered new tserver with Master: ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543)
I20260620 21:58:00.573972 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:57087
I20260620 21:58:00.576283 32173 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:58:00.583755 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.573331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.0.0.1:52114)
0620 21:58:00.573750 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.573792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.573911 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.574499 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.574505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.574946 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:00.575076 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.575085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.576457 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.576460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.577217 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.577226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.578124 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.578127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.578234 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.578790 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:00.578801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:00.580142 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:00.582060 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.582070 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.582076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.582301 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.582593 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.582597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.582599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.582684 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.583138 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:00.583143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:00.583257 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.583390 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.583576 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260620 21:58:00.583775 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.573380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.17.67.193:41543)
0620 21:58:00.574091 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.574116 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.574137 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:58:00.574206 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.574209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.574359 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.574505 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:00.575203 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.576266 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.577336 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.577875 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.578438 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.580274 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:00.580283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:00.580301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:00.580322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:00.581944 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.582411 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.582417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.582418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.582500 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.582779 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.582782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.582784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.583009 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:00.583143 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.583469 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.583634 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":103,"threads_started":1}
I20260620 21:58:00.584738 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet 5711bf89a39c45e88428efb979a28900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:00 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:52114
I20260620 21:58:00.585748 32182 tablet_replica.cc:330] stopping tablet replica
I20260620 21:58:00.586027 32182 raft_consensus.cc:2229] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:00.586274 32182 raft_consensus.cc:2258] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:00.587210 32182 ts_tablet_manager.cc:1905] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:00.590602 32182 ts_tablet_manager.cc:1918] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:00.590715 32182 log.cc:1196] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/5711bf89a39c45e88428efb979a28900
I20260620 21:58:00.590965 32182 ts_tablet_manager.cc:1939] T 5711bf89a39c45e88428efb979a28900 P ce803d93230e4feda73d3213bd06de74: Deleting consensus metadata
I20260620 21:58:00.592675 31970 catalog_manager.cc:4841] TS ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543): tablet 5711bf89a39c45e88428efb979a28900 (table test-table [id=3a5d3011faca468ab1c647fcb7591376]) successfully deleted
W20260620 21:58:00.676599 32177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:00.676869 32177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:00.676988 32177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:00.681450 32177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:00.681510 32177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:00.681622 32177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:58:00.686956 32177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:40327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=44483
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:00.688014 32177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:00.689034 32177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:00.698632 32188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:00.698747 32177 server_base.cc:1034] running on GCE node
W20260620 21:58:00.698562 32187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.698575 32190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:00.699337 32177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:00.700040 32177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:00.701234 32177 hybrid_clock.cc:648] HybridClock initialized: now 1781992680701223 us; error 31 us; skew 500 ppm
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:00.704103 32177 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:58:00.705046 32177 webserver.cc:460] Webserver started at http://127.17.67.194:44483/ using document root <none> and password file <none>
I20260620 21:58:00.705654 32177 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:00.705757 32177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:00.709710 32177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:58:00.711658 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.712873 32177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:58:00.713007 32177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "f643911191884100a8a8d92e0bd01a0f"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "fb71b95d00cfa38065f49555b52d57fc"
server_key_iv: "2819204de17cfcf875d165cfcbc0228d"
server_key_version: "encryptionkey@0"
I20260620 21:58:00.713577 32177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:00.728549 32177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:00.731812 32177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:00.732018 32177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:00.732602 32177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:00.733433 32177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:00.733503 32177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.733593 32177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:00.733650 32177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.742714 32177 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40327
I20260620 21:58:00.742731 32310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40327 every 8 connection(s)
I20260620 21:58:00.743709 32177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:00.744621 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32177
I20260620 21:58:00.745013 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:46315
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=38091
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:00.758447 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.745585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:43311 (local address 127.17.67.254:46181)
0620 21:58:00.745757 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.745763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.746598 (+ 835us) server_negotiation.cc:407] Connection header received
0620 21:58:00.747990 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.747996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.748070 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.748175 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:00.749366 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.750333 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.751505 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.751960 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.752089 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.754604 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:00.754609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:00.754615 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:00.754654 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:00.756413 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.756904 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.756907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.756909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.756995 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.757303 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.757306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.757308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.757521 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:00.757652 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.758141 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.758276 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:00.759302 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.745975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:43311)
0620 21:58:00.746431 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.746483 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.747764 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.748355 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.748362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.748724 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:00.749206 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.749219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.750469 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.750472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.751371 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.751382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.752128 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.752131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.752274 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.752901 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:00.752919 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:00.754410 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:00.756540 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.756545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.756558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.756810 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.757081 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.757083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.757084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.757220 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.757674 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:00.757679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:00.757916 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.758076 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.758354 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":136,"threads_started":1}
I20260620 21:58:00.760480 32311 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:58:00.760731 32311 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:00.761219 32311 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
I20260620 21:58:00.762286 31984 ts_manager.cc:194] Registered new tserver with Master: f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327)
I20260620 21:58:00.763051 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:43311
W20260620 21:58:00.858464 32314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:00.858722 32314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:00.858764 32314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:00.862792 32314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:00.862845 32314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:00.862901 32314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:58:00.868175 32314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:46315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=38091
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:00.869158 32314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:00.870206 32314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:00.879772 32322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.879792 32319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:00.880149 32320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:00.880271 32314 server_base.cc:1034] running on GCE node
I20260620 21:58:00.880654 32314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:00.881363 32314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:00.882537 32314 hybrid_clock.cc:648] HybridClock initialized: now 1781992680882508 us; error 49 us; skew 500 ppm
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:00.885246 32314 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:58:00.886173 32314 webserver.cc:460] Webserver started at http://127.17.67.195:38091/ using document root <none> and password file <none>
I20260620 21:58:00.886720 32314 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:00.886792 32314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:00.890964 32314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:58:00.893162 32329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.894546 32314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:00.894662 32314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "579c3a7f2fb71af75d639a62a6cff06b"
server_key_iv: "e5597307a3162c9f2e189110a0ebe98e"
server_key_version: "encryptionkey@0"
I20260620 21:58:00.895246 32314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:00.907835 32314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:00.910975 32314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:00.911139 32314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:00.911659 32314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:00.912426 32314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:00.912472 32314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.912510 32314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:00.912530 32314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:00.921640 32314 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:46315
I20260620 21:58:00.921711 32442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:46315 every 8 connection(s)
I20260620 21:58:00.922688 32314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:00.932040 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32314
Jun 20 21:58:00 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992680, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:00.937353 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.924533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:56879 (local address 127.17.67.254:46181)
0620 21:58:00.924689 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.924694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.926357 (+ 1663us) server_negotiation.cc:407] Connection header received
0620 21:58:00.927297 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.927300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.927359 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.927446 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:00.928726 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.929273 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.930400 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.930636 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.930727 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.933480 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:00.933483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:00.933485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:00.933523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:00.935224 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.935767 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.935772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.935774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.935855 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.936209 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.936214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.936216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.936382 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:00.936488 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.937085 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.937195 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:58:00.938357 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.925017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:56879)
0620 21:58:00.926037 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.926093 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.927073 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.927611 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.927617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.927972 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:00.928436 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.928448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.929389 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.929392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.930286 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.930295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.930767 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.930770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.930906 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.931513 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:00.931529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:00.933298 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:00.935360 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.935366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.935381 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.935659 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.935959 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:00.935962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:00.935964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:00.936110 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:00.936495 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:00.936503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:00.936836 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.937046 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.937364 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":611,"threads_started":1}
I20260620 21:58:00.939543 32443 heartbeater.cc:347] Connected to a master server at 127.17.67.254:46181
I20260620 21:58:00.939790 32443 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:00.940376 32443 heartbeater.cc:511] Master 127.17.67.254:46181 requested a full tablet report, sending...
I20260620 21:58:00.940698 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.932768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.17.67.254:46181)
0620 21:58:00.932987 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.932993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.933210 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:58:00.933367 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.933370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.933417 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.933524 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:00.934405 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.935070 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.935934 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.936163 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.936270 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.937619 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:00.937622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:00.937625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:00.937648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:00.939049 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.939451 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.939457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.939466 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.939574 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:00.939897 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:00.939900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:00.939902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:00.940105 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:00.940305 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.940429 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.940552 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260620 21:58:00.941481 31984 ts_manager.cc:194] Registered new tserver with Master: 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315)
I20260620 21:58:00.942065 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:56879
I20260620 21:58:00.942413 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:58:00.946534 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.943022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.17.67.254:46181)
0620 21:58:00.943133 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.943138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.943308 (+ 170us) server_negotiation.cc:407] Connection header received
0620 21:58:00.943458 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.943460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.943543 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.943621 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:00.943847 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.944320 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.945182 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.945389 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.945464 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.945818 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:00.946183 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:00.946275 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.946338 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.946383 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:58:00.947134 31984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:00.950516 31984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:00.958966 32107 tablet_service.cc:1467] Processing CreateTablet for tablet dd0499c460944641acd61378be8e73cb (DEFAULT_TABLE table=test-table [id=e1d758c1623c4b529e2c07a865e81d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:00.959609 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0499c460944641acd61378be8e73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:00.962169 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.957152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.0.0.1:41928)
0620 21:58:00.957282 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.957301 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.957387 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.958314 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.958317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.958334 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:00.958441 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.958449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.959673 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.959678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.960567 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.960576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.961727 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.961730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.961829 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.961871 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.961958 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.962039 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:58:00.962219 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.957359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.17.67.194:40327)
0620 21:58:00.957751 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.957765 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.957788 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:58:00.957868 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.957873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.958158 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.958328 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:00.958590 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.959545 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.960691 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.961554 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.961729 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.961876 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.961996 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.962084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":142,"threads_started":1}
I20260620 21:58:00.962457 32065 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Bootstrap starting.
I20260620 21:58:00.963094 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.957639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:46315 (local address 127.0.0.1:60232)
0620 21:58:00.957833 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:00.957847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:00.957932 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:00.958849 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:00.958853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:00.958871 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:00.958952 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.958958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.960268 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.960271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.961340 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:00.961352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.962694 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.962698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:00.962787 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.962810 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:58:00.962888 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:58:00.962956 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 21:58:00.963143 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:00.958028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.17.67.195:46315)
0620 21:58:00.958395 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:00.958413 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:00.958436 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:58:00.958504 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:00.958509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:00.958706 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:00.958859 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:00.959047 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.960107 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.961500 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:00.962521 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:00.962668 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:00.962802 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:00.962917 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:58:00.963007 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":196,"threads_started":1}
I20260620 21:58:00.963382 32065 tablet_bootstrap.cc:654] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:00.964824 32065 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: No bootstrap required, opened a new log
I20260620 21:58:00.964916 32065 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:00.965349 32065 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.965471 32065 raft_consensus.cc:374] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:00.965508 32065 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce803d93230e4feda73d3213bd06de74, State: Initialized, Role: FOLLOWER
I20260620 21:58:00.965667 32065 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.965324 32245 tablet_service.cc:1467] Processing CreateTablet for tablet dd0499c460944641acd61378be8e73cb (DEFAULT_TABLE table=test-table [id=e1d758c1623c4b529e2c07a865e81d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:00.966400 32065 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:00.966441 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0499c460944641acd61378be8e73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:00.966395 32377 tablet_service.cc:1467] Processing CreateTablet for tablet dd0499c460944641acd61378be8e73cb (DEFAULT_TABLE table=test-table [id=e1d758c1623c4b529e2c07a865e81d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:00.967296 32377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0499c460944641acd61378be8e73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:00.973611 32456 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Bootstrap starting.
I20260620 21:58:00.975863 32456 tablet_bootstrap.cc:654] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:00.976356 32457 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Bootstrap starting.
I20260620 21:58:00.976835 32456 log.cc:824] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:00.979152 32456 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: No bootstrap required, opened a new log
I20260620 21:58:00.979205 32457 tablet_bootstrap.cc:654] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:00.979358 32456 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:58:00.980001 32457 log.cc:824] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:00.981992 32457 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: No bootstrap required, opened a new log
I20260620 21:58:00.982189 32457 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:58:00.983202 32456 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.983472 32456 raft_consensus.cc:374] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:00.983518 32456 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e7ac8b4cc34e4b8ea057cc796d0b1c, State: Initialized, Role: FOLLOWER
I20260620 21:58:00.983981 32456 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.984690 32443 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
I20260620 21:58:00.984993 32456 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:58:00.985939 32457 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.986218 32457 raft_consensus.cc:374] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:00.986299 32457 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f643911191884100a8a8d92e0bd01a0f, State: Initialized, Role: FOLLOWER
I20260620 21:58:00.986713 32457 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:00.987357 32311 heartbeater.cc:503] Master 127.17.67.254:46181 was elected leader, sending a full tablet report...
I20260620 21:58:00.987653 32457 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 21:58:00.994561 32312 tablet.cc:2369] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:58:01.044849 32174 tablet.cc:2369] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:01.051528 32460 raft_consensus.cc:482] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:01.051801 32460 raft_consensus.cc:504] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.053224 32460 leader_election.cc:288] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327), ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543)
I20260620 21:58:01.058178 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.053813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48137 (local address 127.17.67.194:40327)
0620 21:58:01.053957 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:01.053962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:01.054282 (+ 320us) server_negotiation.cc:407] Connection header received
0620 21:58:01.054338 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:01.054344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:01.054402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:01.054473 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:01.054803 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.055584 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.057037 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.057591 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.057688 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:01.057728 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:01.057923 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 21:58:01.058014 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:01.058187 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.053542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.17.67.195:48137)
0620 21:58:01.054169 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.054184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:01.054275 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:01.054490 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:01.054492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:01.054527 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:01.054679 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:01.054687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.055743 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.055746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:01.056897 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:01.056910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.057691 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.057695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:01.057791 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:01.057818 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:58:01.057902 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:58:01.057989 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":238,"spinlock_wait_cycles":2816,"thread_start_us":143,"threads_started":1}
I20260620 21:58:01.058434 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.053537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.17.67.195:53763)
0620 21:58:01.053926 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.053942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:01.054088 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:01.054421 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:01.054423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:01.054436 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:01.054686 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:01.054692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.055705 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.055709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:01.057145 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:01.057159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.058016 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.058019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:01.058125 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:01.058148 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:58:01.058238 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:58:01.058312 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":112,"spinlock_wait_cycles":9600}
I20260620 21:58:01.058459 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.053708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:53763 (local address 127.17.67.193:41543)
0620 21:58:01.053877 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:01.053888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:01.054048 (+ 160us) server_negotiation.cc:407] Connection header received
0620 21:58:01.054203 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:01.054205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:01.054286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:01.054394 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:01.054803 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.055581 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.057305 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.057902 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.058006 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:01.058157 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:01.058251 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 21:58:01.058336 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:58:01.059018 32265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f" is_pre_election: true
I20260620 21:58:01.059005 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce803d93230e4feda73d3213bd06de74" is_pre_election: true
I20260620 21:58:01.059334 32265 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 0.
I20260620 21:58:01.059346 32127 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 0.
I20260620 21:58:01.059811 32332 leader_election.cc:302] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e7ac8b4cc34e4b8ea057cc796d0b1c, f643911191884100a8a8d92e0bd01a0f; no voters:
I20260620 21:58:01.060119 32460 raft_consensus.cc:2781] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:01.060201 32460 raft_consensus.cc:482] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:01.060245 32460 raft_consensus.cc:3037] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:01.061215 32460 raft_consensus.cc:504] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.061556 32460 leader_election.cc:288] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 election: Requested vote from peers f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327), ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543)
I20260620 21:58:01.061934 32127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce803d93230e4feda73d3213bd06de74"
I20260620 21:58:01.062001 32265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f"
I20260620 21:58:01.062075 32127 raft_consensus.cc:3037] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:01.062135 32265 raft_consensus.cc:3037] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:01.063100 32265 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 1.
I20260620 21:58:01.063117 32127 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e7ac8b4cc34e4b8ea057cc796d0b1c in term 1.
I20260620 21:58:01.063374 32332 leader_election.cc:302] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e7ac8b4cc34e4b8ea057cc796d0b1c, f643911191884100a8a8d92e0bd01a0f; no voters:
I20260620 21:58:01.063545 32460 raft_consensus.cc:2781] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:01.063786 32460 raft_consensus.cc:686] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 LEADER]: Becoming Leader. State: Replica: 25e7ac8b4cc34e4b8ea057cc796d0b1c, State: Running, Role: LEADER
I20260620 21:58:01.064121 32460 consensus_queue.cc:237] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.067323 31981 catalog_manager.cc:5495] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195). New cstate: current_term: 1 leader_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:01.089452 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.085441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.17.67.195:46315)
0620 21:58:01.085610 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:01.085614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:01.085646 (+ 32us) server_negotiation.cc:407] Connection header received
0620 21:58:01.085805 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:01.085808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:01.085872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:01.085982 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:01.086259 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.086826 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.087871 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:01.088115 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:01.088231 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:01.088572 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:01.089052 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:01.089200 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:01.089242 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 21:58:01.089301 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:58:01.095505 32127 raft_consensus.cc:1261] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Refusing update from remote peer 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:01.095562 32265 raft_consensus.cc:1261] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Refusing update from remote peer 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:01.096262 32463 consensus_queue.cc:1035] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:01.096501 32460 consensus_queue.cc:1035] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:01.103605 32466 mvcc.cc:205] Tried to move back new op lower bound from 7299042021759639552 to 7299042021641379840. Current Snapshot: MvccSnapshot[applied={T|T < 7299042021759639552}]
I20260620 21:58:01.107088 32468 mvcc.cc:205] Tried to move back new op lower bound from 7299042021759639552 to 7299042021641379840. Current Snapshot: MvccSnapshot[applied={T|T < 7299042021759639552}]
I20260620 21:58:01.112959 31981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:01.113128 31981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:01.115517 31981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Sending DeleteTablet for 3 replicas of tablet dd0499c460944641acd61378be8e73cb
I20260620 21:58:01.116283 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0499c460944641acd61378be8e73cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:01 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:52114
I20260620 21:58:01.116348 32377 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0499c460944641acd61378be8e73cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:01 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:60232
I20260620 21:58:01.116675 32474 tablet_replica.cc:330] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: stopping tablet replica
I20260620 21:58:01.116807 32474 raft_consensus.cc:2229] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:01.116835 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0499c460944641acd61378be8e73cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:01 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:41928
I20260620 21:58:01.117000 32474 raft_consensus.cc:2258] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:01.116998 32475 tablet_replica.cc:330] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: stopping tablet replica
I20260620 21:58:01.117363 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 32023
I20260620 21:58:01.117374 32475 raft_consensus.cc:2229] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:01.117827 32475 raft_consensus.cc:2258] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:01.119303 32475 ts_tablet_manager.cc:1905] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:01.122303 32475 ts_tablet_manager.cc:1918] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:01.122469 32475 log.cc:1196] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/dd0499c460944641acd61378be8e73cb
I20260620 21:58:01.122924 32475 ts_tablet_manager.cc:1939] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Deleting consensus metadata
W20260620 21:58:01.124241 31970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:41543 (error 108)
I20260620 21:58:01.124588 32476 tablet_replica.cc:330] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: stopping tablet replica
I20260620 21:58:01.124787 31970 catalog_manager.cc:4841] TS 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315): tablet dd0499c460944641acd61378be8e73cb (table test-table [id=e1d758c1623c4b529e2c07a865e81d09]) successfully deleted
I20260620 21:58:01.124950 32476 raft_consensus.cc:2229] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:01.125329 32476 raft_consensus.cc:2258] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:58:01.125829 32453 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:58:01.125157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.0.0.1:52124)
0620 21:58:01.125294 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.125310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:01.125402 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:01.125704 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:41543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:58:01.125978 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 32177
W20260620 21:58:01.126017 31970 catalog_manager.cc:4584] TS ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd0499c460944641acd61378be8e73cb: Network error: Client connection negotiation failed: client connection to 127.17.67.193:41543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:58:01.132290 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 32314
I20260620 21:58:01.133095 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.132399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.0.0.1:41930)
0620 21:58:01.132920 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.133001 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.194:40327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
I20260620 21:58:01.137959 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 31952
I20260620 21:58:01.144670 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--webserver_interface=127.17.67.254
--webserver_port=34589
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:01.261327 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:01.261657 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:01.261756 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:01.265898 32478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:01.265967 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:01.265987 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:01.266006 32478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:01.266022 32478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:01.271695 32478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:46181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:01.272867 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:01.274042 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:01.283308 32483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.283344 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.283308 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.283535 32478 server_base.cc:1034] running on GCE node
I20260620 21:58:01.284227 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:01.285420 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:01.286633 32478 hybrid_clock.cc:648] HybridClock initialized: now 1781992681286605 us; error 41 us; skew 500 ppm
Jun 20 21:58:01 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:01.289438 32478 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:58:01.290437 32478 webserver.cc:460] Webserver started at http://127.17.67.254:34589/ using document root <none> and password file <none>
I20260620 21:58:01.291038 32478 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:01.291105 32478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:01.294996 32478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 21:58:01.297144 32493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.298520 32478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 21:58:01.298648 32478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "9ec51dfc56ca46d8913cc86e61ec49b7"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "2cedafb23e74df210f459efa74213156"
server_key_iv: "8c1ea5010b765e4789ef1ca352f7eaa1"
server_key_version: "encryptionkey@0"
I20260620 21:58:01.299197 32478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:01.326243 32478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:01.329308 32478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:01.329474 32478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:01.337057 32478 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:46181
I20260620 21:58:01.337414 32545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:46181 every 8 connection(s)
I20260620 21:58:01.338272 32478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:01.341826 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32478
I20260620 21:58:01.342368 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:41543
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=41915
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:01.343407 32546 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:58:01.345472 32546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap starting.
I20260620 21:58:01.364451 32546 log.cc:824] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:01.376744 32546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:01.377210 32546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7: Bootstrap complete.
I20260620 21:58:01.381712 32546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:01.382160 32546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Initialized, Role: FOLLOWER
I20260620 21:58:01.382781 32546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:01.382920 32546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:01.383019 32546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:01.383149 32546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:58:01.385273 32546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:01.385712 32546 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec51dfc56ca46d8913cc86e61ec49b7; no voters:
I20260620 21:58:01.386086 32546 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:58:01.386183 32550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:58:01.386384 32550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [term 3 LEADER]: Becoming Leader. State: Replica: 9ec51dfc56ca46d8913cc86e61ec49b7, State: Running, Role: LEADER
I20260620 21:58:01.386718 32550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } }
I20260620 21:58:01.387481 32546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:01.387856 32550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec51dfc56ca46d8913cc86e61ec49b7. Latest consensus state: current_term: 3 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:58:01.387985 32550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:01.388348 32551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec51dfc56ca46d8913cc86e61ec49b7" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46181 } } }
I20260620 21:58:01.388523 32551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec51dfc56ca46d8913cc86e61ec49b7 [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:01.388370 32554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:01.394305 32554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:01.395047 32554 catalog_manager.cc:1259] Loaded cluster ID: 8b86a3577f4b4bd18553f5719f07bace
I20260620 21:58:01.395165 32554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 21:58:01.396250 32554 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f759b1a79ad google::LogMessage::Fail() at ??:0
@ 0x7f759b1aa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f759b1a74de google::LogMessage::Flush() at ??:0
@ 0x7f759b1a8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f75a5ae6fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f75a5ae7d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f75a5ae4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f75a5b28640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f759d3f93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f759d3f9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f759d3fb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f759d3ea512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f759ea2e6db start_thread at ??:0
@ 0x7f7599ca561f clone at ??:0
W20260620 21:58:01.478807 32548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:01.479048 32548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:01.479091 32548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:01.483295 32548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:01.483381 32548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:01.483501 32548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:58:01.489040 32548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:41543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=41915
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:01.490224 32548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:01.491309 32548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:01.501401 32588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.501397 32590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.501704 32548 server_base.cc:1034] running on GCE node
W20260620 21:58:01.501919 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.502398 32548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:01.503223 32548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:01.504428 32548 hybrid_clock.cc:648] HybridClock initialized: now 1781992681504391 us; error 45 us; skew 500 ppm
Jun 20 21:58:01 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:01.508086 32548 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:58:01.509253 32548 webserver.cc:460] Webserver started at http://127.17.67.193:41915/ using document root <none> and password file <none>
I20260620 21:58:01.510068 32548 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:01.510171 32548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:01.515360 32548 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:58:01.518054 32597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.519753 32548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:01.519871 32548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "ce803d93230e4feda73d3213bd06de74"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "7e00e19784245446972583ce3639038f"
server_key_iv: "7a112f9398d9b884a392c9b922a53c9a"
server_key_version: "encryptionkey@0"
I20260620 21:58:01.520567 32548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:01.550166 32548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:01.554636 32548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:01.554889 32548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:01.555800 32548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:01.557273 32604 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:58:01.561617 32548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:58:01.561767 32548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 21:58:01.561854 32548 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:58:01.564116 32548 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:58:01.564180 32548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:58:01.564560 32604 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Bootstrap starting.
I20260620 21:58:01.575462 32548 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:41543
I20260620 21:58:01.575809 32711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:41543 every 8 connection(s)
I20260620 21:58:01.576709 32548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:01.580354 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32548
I20260620 21:58:01.580827 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:40327
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=44483
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:01.585446 32604 log.cc:824] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:01.595899 32604 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:01.596343 32604 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Bootstrap complete.
I20260620 21:58:01.597810 32604 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20260620 21:58:01.601023 32604 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.601558 32604 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce803d93230e4feda73d3213bd06de74, State: Initialized, Role: FOLLOWER
I20260620 21:58:01.602165 32604 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.603314 32604 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260620 21:58:01.655838 32714 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:58:01.579107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:52059)
0620 21:58:01.579719 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.579779 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:01.581294 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:01.655045 (+ 73751us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":170,"threads_started":1}
W20260620 21:58:01.656538 32712 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:46181 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:46181: Client connection negotiation failed: client connection to 127.17.67.254:46181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:58:01.657294 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.657033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:52423)
0620 21:58:01.657163 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.657241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:58:01.658088 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.657839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:43171)
0620 21:58:01.657973 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.658029 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:58:01.658286 32712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:58:01.705093 32715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:01.705379 32715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:01.705425 32715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:01.709748 32715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:01.709829 32715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:01.709918 32715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:58:01.715245 32715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:40327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=44483
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:01.716287 32715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:01.717255 32715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:01.727231 32725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.727339 32715 server_base.cc:1034] running on GCE node
W20260620 21:58:01.727205 32722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.727231 32723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.728027 32715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:01.728672 32715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:01.729830 32715 hybrid_clock.cc:648] HybridClock initialized: now 1781992681729802 us; error 40 us; skew 500 ppm
Jun 20 21:58:01 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:01.732669 32715 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:58:01.733704 32715 webserver.cc:460] Webserver started at http://127.17.67.194:44483/ using document root <none> and password file <none>
I20260620 21:58:01.734280 32715 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:01.734375 32715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:01.738477 32715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:58:01.740924 32732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.742357 32715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:58:01.742514 32715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "f643911191884100a8a8d92e0bd01a0f"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "fb71b95d00cfa38065f49555b52d57fc"
server_key_iv: "2819204de17cfcf875d165cfcbc0228d"
server_key_version: "encryptionkey@0"
I20260620 21:58:01.743101 32715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:01.773900 32715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:01.777832 32715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:01.778108 32715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:01.778767 32715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:01.779934 32739 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:58:01.784433 32715 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:58:01.784556 32715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 21:58:01.784653 32715 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:58:01.786594 32715 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:58:01.786675 32715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260620 21:58:01.786953 32739 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Bootstrap starting.
I20260620 21:58:01.797276 32715 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:40327
I20260620 21:58:01.797353 378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:40327 every 8 connection(s)
I20260620 21:58:01.798363 32715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:01.800642 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 32715
I20260620 21:58:01.801198 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:46315
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:46181
--webserver_port=38091
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:01.802619 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.800524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:33821)
0620 21:58:01.801129 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.801611 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":167,"threads_started":1}
W20260620 21:58:01.803164 379 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:46181 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:46181: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
I20260620 21:58:01.803778 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.803549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:43383)
0620 21:58:01.803667 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.803720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 21:58:01.804417 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.804214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:37065)
0620 21:58:01.804312 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.804358 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:58:01.804648 379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:58:01.807739 32739 log.cc:824] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:01.817503 32739 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:01.817960 32739 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Bootstrap complete.
I20260620 21:58:01.819459 32739 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20260620 21:58:01.823417 32739 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.823972 32739 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f643911191884100a8a8d92e0bd01a0f, State: Initialized, Role: FOLLOWER
I20260620 21:58:01.824488 32739 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:01.825517 32739 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:58:01.825793 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.825463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:54201)
0620 21:58:01.825632 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.825703 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 21:58:01.826534 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:01.826283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:47347)
0620 21:58:01.826423 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:01.826478 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:58:01.924242 382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:01.924496 382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:01.924541 382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:01.928714 382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:01.928774 382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:01.928860 382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:58:01.934271 382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:46315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=38091
--tserver_master_addrs=127.17.67.254:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:01.935354 382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:01.936386 382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:01.946450 392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.946450 389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:01.946484 390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:01.946808 382 server_base.cc:1034] running on GCE node
I20260620 21:58:01.947391 382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:01.948135 382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:01.949339 382 hybrid_clock.cc:648] HybridClock initialized: now 1781992681949329 us; error 33 us; skew 500 ppm
Jun 20 21:58:01 dist-test-slave-cx8d krb5kdc[31405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:01.952668 382 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:58:01.953763 382 webserver.cc:460] Webserver started at http://127.17.67.195:38091/ using document root <none> and password file <none>
I20260620 21:58:01.954383 382 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:01.954491 382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:01.958706 382 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:58:01.960973 399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.962334 382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:01.962579 382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c"
format_stamp: "Formatted at 2026-06-20 21:57:59 on dist-test-slave-cx8d"
server_key: "579c3a7f2fb71af75d639a62a6cff06b"
server_key_iv: "e5597307a3162c9f2e189110a0ebe98e"
server_key_version: "encryptionkey@0"
I20260620 21:58:01.963248 382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:01.982426 382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:01.985636 382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:01.985842 382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:01.986452 382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:01.987294 382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:01.987375 382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.987469 382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:01.987519 382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:01.996655 382 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:46315
I20260620 21:58:01.996678 512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:46315 every 8 connection(s)
I20260620 21:58:01.997738 382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:01.999753 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 382
I20260620 21:58:02.002310 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:02.000365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:54347)
0620 21:58:02.000898 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:02.001034 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":176,"threads_started":1}
W20260620 21:58:02.002797 513 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:46181 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:46181: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
I20260620 21:58:02.003538 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:02.003252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:56319)
0620 21:58:02.003376 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:02.003448 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 21:58:02.004280 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:02.004098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:33343)
0620 21:58:02.004197 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:02.004238 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:58:02.004443 513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:58:02.659394 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:02.658939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:42193)
0620 21:58:02.659218 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:02.659283 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260620 21:58:02.827840 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:02.827395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:44187)
0620 21:58:02.827665 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:02.827745 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260620 21:58:03.005532 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.005002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:41189)
0620 21:58:03.005333 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.005423 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260620 21:58:03.116290 520 raft_consensus.cc:482] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:03.116485 520 raft_consensus.cc:504] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:03.117743 520 leader_election.cc:288] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327), 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315)
Jun 20 21:58:03 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.195@KRBTEST.COM
Jun 20 21:58:03 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.194@KRBTEST.COM
I20260620 21:58:03.133080 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.118041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:46315 (local address 127.17.67.193:56359)
0620 21:58:03.118591 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.118609 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:03.118751 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:03.120112 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:03.120115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:03.120648 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:03.121155 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:03.121170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.122529 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.122532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:03.124171 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:03.124183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.124972 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.124975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:03.125124 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:03.125841 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:03.125857 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:03.127724 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:03.130483 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:03.130488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:03.130500 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:03.130767 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:03.131159 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:03.131163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:03.131320 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:03.132079 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:03.132085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:03.132332 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 21:58:03.132556 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 21:58:03.132872 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":242,"spinlock_wait_cycles":18816,"thread_start_us":109,"threads_started":1}
I20260620 21:58:03.133147 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.117955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:40327 (local address 127.17.67.193:59895)
0620 21:58:03.118899 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.118909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:03.119017 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:03.120035 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:03.120042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:03.120649 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:03.121155 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:03.121171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.122628 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.122631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:03.124171 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:03.124183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.126110 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.126114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:03.126201 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:03.126663 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:03.126676 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:03.128130 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:03.131162 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:03.131166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:03.131396 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:03.131774 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:03.131780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:03.131868 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:03.132582 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:03.132586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:03.132691 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 21:58:03.132811 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 21:58:03.132961 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":185,"spinlock_wait_cycles":5888}
I20260620 21:58:03.133447 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.118164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59895 (local address 127.17.67.194:40327)
0620 21:58:03.118698 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:03.118714 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:03.119009 (+ 295us) server_negotiation.cc:407] Connection header received
0620 21:58:03.119284 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:03.119289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:03.119658 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:03.119986 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:03.121328 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.122484 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.125231 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.125982 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.126120 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:03.128302 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:03.128312 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:03.128334 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:03.128375 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:03.130767 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131523 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:03.131529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:03.131535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:03.131658 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131968 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:03.131973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:03.131975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:03.132453 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:03.132587 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:03.132955 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 21:58:03.133282 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":106,"threads_started":1}
I20260620 21:58:03.133109 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.118192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:56359 (local address 127.17.67.195:46315)
0620 21:58:03.118519 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:03.118541 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:03.118757 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:58:03.119557 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:03.119563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:03.119851 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:03.120172 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:03.121344 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.122377 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.124307 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:03.124845 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:03.125044 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:03.127968 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:03.127974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:03.127989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:03.128020 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:03.130332 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:03.130900 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:03.130906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:03.130913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:03.131051 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:03.131470 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:03.131475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:03.131482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:03.131953 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:03.132107 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:03.132672 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 21:58:03.132947 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260620 21:58:03.134794 333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "ce803d93230e4feda73d3213bd06de74" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f" is_pre_election: true
I20260620 21:58:03.135154 333 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce803d93230e4feda73d3213bd06de74 in term 1.
I20260620 21:58:03.136250 32600 leader_election.cc:302] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce803d93230e4feda73d3213bd06de74, f643911191884100a8a8d92e0bd01a0f; no voters:
W20260620 21:58:03.136461 32599 leader_election.cc:341] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315): Not found: Tablet not found: dd0499c460944641acd61378be8e73cb
I20260620 21:58:03.136567 520 raft_consensus.cc:2781] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:58:03.136641 520 raft_consensus.cc:482] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:03.136687 520 raft_consensus.cc:3037] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:58:03.139058 520 raft_consensus.cc:504] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:03.139456 520 leader_election.cc:288] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 election: Requested vote from peers f643911191884100a8a8d92e0bd01a0f (127.17.67.194:40327), 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315)
I20260620 21:58:03.139917 333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0499c460944641acd61378be8e73cb" candidate_uuid: "ce803d93230e4feda73d3213bd06de74" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f643911191884100a8a8d92e0bd01a0f"
I20260620 21:58:03.140043 333 raft_consensus.cc:3037] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 1 FOLLOWER]: Advancing to term 2
W20260620 21:58:03.140321 32599 leader_election.cc:341] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315): Not found: Tablet not found: dd0499c460944641acd61378be8e73cb
I20260620 21:58:03.142395 333 raft_consensus.cc:2454] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce803d93230e4feda73d3213bd06de74 in term 2.
I20260620 21:58:03.142755 32600 leader_election.cc:302] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ce803d93230e4feda73d3213bd06de74, f643911191884100a8a8d92e0bd01a0f; no voters: 25e7ac8b4cc34e4b8ea057cc796d0b1c
I20260620 21:58:03.142951 520 raft_consensus.cc:2781] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:58:03.143160 520 raft_consensus.cc:686] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [term 2 LEADER]: Becoming Leader. State: Replica: ce803d93230e4feda73d3213bd06de74, State: Running, Role: LEADER
I20260620 21:58:03.143486 520 consensus_queue.cc:237] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:03.143949 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.143679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:55093)
0620 21:58:03.143811 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.143873 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 21:58:03.717736 333 raft_consensus.cc:1261] T dd0499c460944641acd61378be8e73cb P f643911191884100a8a8d92e0bd01a0f [term 2 FOLLOWER]: Refusing update from remote peer ce803d93230e4feda73d3213bd06de74: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 21:58:03.718495 526 consensus_queue.cc:1035] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 21:58:03.720842 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.720457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:51827)
0620 21:58:03.720686 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.720744 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":64,"threads_started":1}
W20260620 21:58:03.723812 32599 consensus_peers.cc:487] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74 -> Peer 25e7ac8b4cc34e4b8ea057cc796d0b1c (127.17.67.195:46315): Couldn't send request to peer 25e7ac8b4cc34e4b8ea057cc796d0b1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd0499c460944641acd61378be8e73cb. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:58:03.723831 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.723579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:41099)
0620 21:58:03.723720 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.723761 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 21:58:03.723842 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:03.723264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:54557)
0620 21:58:03.723673 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:03.723741 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":112,"threads_started":1}
I20260620 21:58:04.006920 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.006492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:35111)
0620 21:58:04.006765 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.006821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
I20260620 21:58:04.194351 536 ts_tablet_manager.cc:927] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Initiating tablet copy from peer ce803d93230e4feda73d3213bd06de74 (127.17.67.193:41543)
I20260620 21:58:04.194948 536 tablet_copy_client.cc:316] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet copy: Beginning tablet copy session from remote peer at address 127.17.67.193:41543
Jun 20 21:58:04 dist-test-slave-cx8d krb5kdc[31405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992681, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:58:04.209853 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.195457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:41543 (local address 127.17.67.195:46183)
0620 21:58:04.195610 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.195643 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:04.195759 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:04.196453 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:04.196459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:04.196865 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:04.197039 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:04.197046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:04.198310 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:04.198314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:04.199075 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:04.199086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:04.200056 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:04.200059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:04.200147 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:04.204223 (+ 4076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:04.204237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:04.205958 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:04.207909 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:04.207919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:04.207926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:04.208168 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:04.208529 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:04.208532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:04.208534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:04.208623 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:04.209129 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:04.209136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:04.209272 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 21:58:04.209422 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 21:58:04.209662 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 21:58:04.209842 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.195669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:46183 (local address 127.17.67.193:41543)
0620 21:58:04.195997 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:04.196006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:04.196026 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:58:04.196070 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:04.196075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:04.196257 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:04.196386 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:04.197206 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:04.198160 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:04.199258 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:04.199894 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:04.199985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:04.206166 (+ 6181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:04.206177 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:04.206190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:04.206218 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:04.207781 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:04.208307 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:04.208313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:04.208314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:04.208432 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:04.208723 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:04.208726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:04.208728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:04.209004 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:04.209161 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:04.209508 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 21:58:04.209690 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
I20260620 21:58:04.211097 32686 tablet_copy_service.cc:140] P ce803d93230e4feda73d3213bd06de74: Received BeginTabletCopySession request for tablet dd0499c460944641acd61378be8e73cb from peer 25e7ac8b4cc34e4b8ea057cc796d0b1c ({username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:46183)
I20260620 21:58:04.211323 32686 tablet_copy_service.cc:161] P ce803d93230e4feda73d3213bd06de74: Beginning new tablet copy session on tablet dd0499c460944641acd61378be8e73cb from peer 25e7ac8b4cc34e4b8ea057cc796d0b1c at {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:46183: session id = 25e7ac8b4cc34e4b8ea057cc796d0b1c-dd0499c460944641acd61378be8e73cb
I20260620 21:58:04.212621 32686 tablet_copy_source_session.cc:205] T dd0499c460944641acd61378be8e73cb P ce803d93230e4feda73d3213bd06de74: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:58:04.216728 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0499c460944641acd61378be8e73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:04.223260 536 tablet_copy_client.cc:791] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet copy: Starting download of 0 data blocks...
I20260620 21:58:04.223486 536 tablet_copy_client.cc:655] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet copy: Starting download of 1 WAL segments...
I20260620 21:58:04.225386 536 tablet_copy_client.cc:531] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:58:04.227106 536 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Bootstrap starting.
I20260620 21:58:04.232412 536 log.cc:824] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:04.239015 536 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:04.239327 536 tablet_bootstrap.cc:492] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Bootstrap complete.
I20260620 21:58:04.239902 536 ts_tablet_manager.cc:1397] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20260620 21:58:04.242394 536 raft_consensus.cc:348] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:04.242664 536 raft_consensus.cc:729] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e7ac8b4cc34e4b8ea057cc796d0b1c, State: Initialized, Role: FOLLOWER
I20260620 21:58:04.243151 536 consensus_queue.cc:260] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f643911191884100a8a8d92e0bd01a0f" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 40327 } } peers { permanent_uuid: "25e7ac8b4cc34e4b8ea057cc796d0b1c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 46315 } } peers { permanent_uuid: "ce803d93230e4feda73d3213bd06de74" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 41543 } }
I20260620 21:58:04.244197 536 ts_tablet_manager.cc:1428] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260620 21:58:04.244470 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.244168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:35063)
0620 21:58:04.244292 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.244354 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:58:04.245015 32686 tablet_copy_service.cc:342] P ce803d93230e4feda73d3213bd06de74: Request end of tablet copy session 25e7ac8b4cc34e4b8ea057cc796d0b1c-dd0499c460944641acd61378be8e73cb received from {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:46183
I20260620 21:58:04.245150 32686 tablet_copy_service.cc:433] P ce803d93230e4feda73d3213bd06de74: ending tablet copy session 25e7ac8b4cc34e4b8ea057cc796d0b1c-dd0499c460944641acd61378be8e73cb on tablet dd0499c460944641acd61378be8e73cb with peer 25e7ac8b4cc34e4b8ea057cc796d0b1c
I20260620 21:58:04.245203 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.244967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:34231)
0620 21:58:04.245071 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.245121 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 21:58:04.725091 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.724626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:35003)
0620 21:58:04.724898 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.724967 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260620 21:58:04.725136 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.724627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:35997)
0620 21:58:04.724989 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.725042 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":108,"threads_started":1}
I20260620 21:58:04.808902 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.808571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:51033)
0620 21:58:04.808716 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.808793 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:58:04.808935 467 raft_consensus.cc:1203] T dd0499c460944641acd61378be8e73cb P 25e7ac8b4cc34e4b8ea057cc796d0b1c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 21:58:04.810290 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:04.809872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:37669)
0620 21:58:04.810113 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:04.810189 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260620 21:58:05.726480 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:05.726030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:47835)
0620 21:58:05.726321 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:05.726382 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260620 21:58:05.810199 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:05.809768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:42769)
0620 21:58:05.810044 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:05.810107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260620 21:58:05.811419 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:05.811033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:41321)
0620 21:58:05.811290 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:05.811341 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
I20260620 21:58:06.727855 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:06.727389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:50687)
0620 21:58:06.727685 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:06.727750 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260620 21:58:06.811776 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:06.811285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:43899)
0620 21:58:06.811607 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:06.811677 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260620 21:58:06.812604 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:06.812351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:48041)
0620 21:58:06.812480 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:06.812546 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:58:06.812783 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:06.812332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:56755)
0620 21:58:06.812597 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:06.812679 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
I20260620 21:58:07.729202 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:07.728777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:45663)
0620 21:58:07.729046 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:07.729106 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
I20260620 21:58:07.813910 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:07.813384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:51091)
0620 21:58:07.813666 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:07.813736 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
I20260620 21:58:07.814466 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:07.813735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:54921)
0620 21:58:07.813953 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:07.814026 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260620 21:58:08.730471 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:08.730028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:40951)
0620 21:58:08.730301 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:08.730363 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260620 21:58:08.815236 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:08.814776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:37479)
0620 21:58:08.815088 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:08.815145 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260620 21:58:08.815646 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:08.815213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:36989)
0620 21:58:08.815519 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:08.815570 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260620 21:58:09.731755 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:09.731325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:54545)
0620 21:58:09.731591 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:09.731656 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260620 21:58:09.816476 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:09.816049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:47997)
0620 21:58:09.816326 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:09.816384 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
I20260620 21:58:09.816807 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:09.816365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:47513)
0620 21:58:09.816663 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:09.816727 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":56,"threads_started":1}
I20260620 21:58:10.732913 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:10.732519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:50015)
0620 21:58:10.732757 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:10.732815 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260620 21:58:10.817829 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:10.817341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:33647)
0620 21:58:10.817632 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:10.817707 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":111,"threads_started":1}
I20260620 21:58:10.818141 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:10.817614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:44639)
0620 21:58:10.817966 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:10.818039 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260620 21:58:11.734244 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:11.733793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:33285)
0620 21:58:11.734091 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:11.734149 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260620 21:58:11.819288 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:11.818766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:45927)
0620 21:58:11.819096 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:11.819186 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260620 21:58:11.819397 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:11.818886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:49591)
0620 21:58:11.819211 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:11.819270 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260620 21:58:12.735805 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:12.735376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:47037)
0620 21:58:12.735651 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:12.735712 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260620 21:58:12.820626 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:12.820129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:53941)
0620 21:58:12.820427 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:12.820501 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260620 21:58:12.820842 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:12.820370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:54619)
0620 21:58:12.820637 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:12.820733 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260620 21:58:13.736979 574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:13.736586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:47113)
0620 21:58:13.736832 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:13.736885 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260620 21:58:13.822057 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:13.821497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:40509)
0620 21:58:13.821835 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:13.821919 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260620 21:58:13.822223 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:13.821741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:33599)
0620 21:58:13.822050 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:13.822107 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":148,"threads_started":1}
I20260620 21:58:14.738288 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:14.737881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:46885)
0620 21:58:14.738134 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:14.738193 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260620 21:58:14.823446 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:14.822950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:54749)
0620 21:58:14.823255 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:14.823342 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260620 21:58:14.823474 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:14.823046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:32913)
0620 21:58:14.823327 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:14.823385 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260620 21:58:15.739511 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:15.739101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:47577)
0620 21:58:15.739355 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:15.739415 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260620 21:58:15.824617 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:15.824192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:51737)
0620 21:58:15.824473 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:15.824528 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 21:58:15.824640 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:15.824195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:57339)
0620 21:58:15.824505 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:15.824556 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260620 21:58:16.740958 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:16.740370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.194:58095)
0620 21:58:16.740806 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:16.740865 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":95,"threads_started":1}
I20260620 21:58:16.825997 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:16.825455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.195:39585)
0620 21:58:16.825749 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:16.825819 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":135,"threads_started":1}
I20260620 21:58:16.826293 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:16.825723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:46181 (local address 127.17.67.193:32955)
0620 21:58:16.826116 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:16.826193 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:46181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":250,"threads_started":1}
I20260620 21:58:17.011510 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 32548
I20260620 21:58:17.017457 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 32715
I20260620 21:58:17.023988 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 382
2026-06-20T21:58:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19214 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:58:17 dist-test-slave-cx8d krb5kdc[588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:58:17 dist-test-slave-cx8d krb5kdc[588](info): set up 2 sockets
Jun 20 21:58:17 dist-test-slave-cx8d krb5kdc[588](info): commencing operation
krb5kdc: starting...
W20260620 21:58:18.081889 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:58:18.208756 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:18.321969 601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:18.322252 601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:18.322293 601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:18.326514 601 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:58:18.326572 601 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:58:18.326597 601 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:58:18.326612 601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:18.326680 601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:18.326714 601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:18.326754 601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:18.326794 601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:18.332242 601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:18.333343 601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:18.334515 601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:18.343747 609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:18.343963 601 server_base.cc:1034] running on GCE node
W20260620 21:58:18.343840 607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:18.343747 606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:18.344867 601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:18.346138 601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:18.347347 601 hybrid_clock.cc:648] HybridClock initialized: now 1781992698347318 us; error 47 us; skew 500 ppm
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:18.350494 601 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:58:18.351543 601 webserver.cc:460] Webserver started at http://127.17.67.254:39005/ using document root <none> and password file <none>
I20260620 21:58:18.352159 601 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:18.352257 601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:18.352496 601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:18.354628 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "abab11eeb87149358d20d72b2ec37aea"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "f0ccb1eedae0f2bfcbe83abc02c127f4"
server_key_iv: "a21eb84b3a7762be08677f5f844bc6b4"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.355198 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "abab11eeb87149358d20d72b2ec37aea"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "f0ccb1eedae0f2bfcbe83abc02c127f4"
server_key_iv: "a21eb84b3a7762be08677f5f844bc6b4"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.358965 601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:58:18.361392 616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.362726 601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:18.362876 601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "abab11eeb87149358d20d72b2ec37aea"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "f0ccb1eedae0f2bfcbe83abc02c127f4"
server_key_iv: "a21eb84b3a7762be08677f5f844bc6b4"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.362996 601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:18.382784 601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:18.385962 601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:18.386169 601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:18.393505 601 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44089
I20260620 21:58:18.393800 668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44089 every 8 connection(s)
I20260620 21:58:18.394529 601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:18.395268 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 601
I20260620 21:58:18.395426 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:58:18.395718 17679 external_mini_cluster.cc:1428] Setting key dae69bc4f0cad895e1c2109628eb0dde
I20260620 21:58:18.397794 669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:18.404650 669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap starting.
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:18.407332 669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:18.408241 669 log.cc:824] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:18.410470 669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: No bootstrap required, opened a new log
I20260620 21:58:18.411159 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:18.397159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.17.67.254:44089)
0620 21:58:18.397630 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:18.397649 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:18.397693 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:58:18.398546 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:18.398570 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:18.398995 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:18.399364 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:18.400074 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.401648 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.402555 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.403252 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.403427 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:18.406077 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:18.406084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:18.406099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:18.406131 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:18.408120 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.408651 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.408659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.408666 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.408803 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.409125 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.409129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.409130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.409527 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:18.409780 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:18.410064 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 21:58:18.410354 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
I20260620 21:58:18.414466 669 raft_consensus.cc:348] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:18.414711 669 raft_consensus.cc:374] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:18.414805 669 raft_consensus.cc:729] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Initialized, Role: FOLLOWER
I20260620 21:58:18.415277 669 consensus_queue.cc:260] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:18.415431 669 raft_consensus.cc:388] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:18.415516 669 raft_consensus.cc:482] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:18.415627 669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:18.416851 669 raft_consensus.cc:504] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:18.417217 669 leader_election.cc:302] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abab11eeb87149358d20d72b2ec37aea; no voters:
I20260620 21:58:18.417646 669 leader_election.cc:288] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:58:18.417768 674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:18.418069 674 raft_consensus.cc:686] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 LEADER]: Becoming Leader. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Running, Role: LEADER
I20260620 21:58:18.418393 674 consensus_queue.cc:237] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:18.418903 669 sys_catalog.cc:564] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:18.420014 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader abab11eeb87149358d20d72b2ec37aea. Latest consensus state: current_term: 1 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:58:18.420146 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:18.420542 683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:18.420542 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:58:18.421625 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:18.423691 683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:18.429176 683 catalog_manager.cc:1347] Generated new cluster ID: 1b02d91c4cd54ffd959202b75ae04f6e
I20260620 21:58:18.429268 683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:18.450495 683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:58:18.451519 683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:18.462810 683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Generated new TSK 0
I20260620 21:58:18.463833 683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:58:18.501857 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:18.616263 695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:18.616532 695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:18.616608 695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:18.621179 695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:18.621258 695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:18.621321 695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:58:18.626899 695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:18.627995 695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:18.629022 695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:18.639431 703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:18.639541 700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:18.639462 701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:18.639706 695 server_base.cc:1034] running on GCE node
I20260620 21:58:18.640286 695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:18.640980 695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:18.642191 695 hybrid_clock.cc:648] HybridClock initialized: now 1781992698642171 us; error 31 us; skew 500 ppm
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:18.645218 695 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:58:18.646265 695 webserver.cc:460] Webserver started at http://127.17.67.193:36699/ using document root <none> and password file <none>
I20260620 21:58:18.646906 695 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:18.647017 695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:18.647270 695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:18.649530 695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "b907c73706b14b8180da1da77aaa181b"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "64ac235e269334a1f4be282e9a3475de"
server_key_iv: "460272550ff3db30110d0066a68796c1"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.650214 695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "b907c73706b14b8180da1da77aaa181b"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "64ac235e269334a1f4be282e9a3475de"
server_key_iv: "460272550ff3db30110d0066a68796c1"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.654243 695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:58:18.656746 710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.658143 695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:18.658284 695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "b907c73706b14b8180da1da77aaa181b"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "64ac235e269334a1f4be282e9a3475de"
server_key_iv: "460272550ff3db30110d0066a68796c1"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.658406 695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:18.680500 695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:18.683938 695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:18.684180 695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:18.684857 695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:18.685849 695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:18.685930 695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.686009 695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:18.686065 695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.695597 695 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:42239
I20260620 21:58:18.695628 823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:42239 every 8 connection(s)
I20260620 21:58:18.696540 695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:18.698662 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 695
I20260620 21:58:18.698798 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:58:18.699110 17679 external_mini_cluster.cc:1428] Setting key 4e8609740cb91e8bde940204b01e5ff4
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:58:18.712466 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:18.698531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:59823 (local address 127.17.67.254:44089)
0620 21:58:18.698670 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:18.698674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:18.699732 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 21:58:18.701369 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:18.701376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:18.701437 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:18.701530 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:18.702818 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.703503 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.705137 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.705504 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.705642 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:18.708579 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:18.708582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:18.708585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:18.708617 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:18.710176 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.710693 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.710698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.710700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.710781 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.711192 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.711196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.711199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.711419 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:18.711531 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:18.712147 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 21:58:18.712320 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:18.713548 826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:18.699034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:59823)
0620 21:58:18.699583 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:18.699642 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:18.701155 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:18.701744 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:18.701751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:18.702169 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:18.702661 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:18.702674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.703659 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.703662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:18.705018 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:18.705031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.705698 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.705705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:18.705957 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:18.706671 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:18.706699 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:18.708413 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:18.710296 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:18.710302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:18.710317 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:18.710587 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:18.710886 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:18.710890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:18.710893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:18.711091 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:18.711531 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:18.711539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:18.711862 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 21:58:18.712067 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 21:58:18.712407 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":160,"threads_started":1}
I20260620 21:58:18.714941 824 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:18.715191 824 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:18.715703 824 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:18.717201 633 ts_manager.cc:194] Registered new tserver with Master: b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239)
I20260620 21:58:18.718403 633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:59823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:58:18.729314 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:18.845664 829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:18.845916 829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:18.845959 829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:18.849967 829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:18.850023 829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:18.850081 829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:58:18.855269 829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:18.856282 829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:18.857232 829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:18.866870 837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:18.866891 834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:18.867143 829 server_base.cc:1034] running on GCE node
W20260620 21:58:18.866873 835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:18.867702 829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:18.868413 829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:18.869637 829 hybrid_clock.cc:648] HybridClock initialized: now 1781992698869603 us; error 64 us; skew 500 ppm
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:18.872762 829 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:58:18.873780 829 webserver.cc:460] Webserver started at http://127.17.67.194:40069/ using document root <none> and password file <none>
I20260620 21:58:18.874423 829 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:18.874521 829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:18.874765 829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:18.877012 829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "fc6c0b465067c934c739780f106cdf4a"
server_key_iv: "cfdd82c91d1ee704effbe3e00741af25"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.877701 829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "fc6c0b465067c934c739780f106cdf4a"
server_key_iv: "cfdd82c91d1ee704effbe3e00741af25"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.881662 829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 21:58:18.884240 844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.885506 829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:58:18.885696 829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "fc6c0b465067c934c739780f106cdf4a"
server_key_iv: "cfdd82c91d1ee704effbe3e00741af25"
server_key_version: "encryptionkey@0"
I20260620 21:58:18.885797 829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:18.903553 829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:18.906842 829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:18.907008 829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:18.907565 829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:18.908363 829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:18.908418 829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.908460 829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:18.908475 829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:18.917974 829 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:39975
I20260620 21:58:18.918068 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:39975 every 8 connection(s)
I20260620 21:58:18.918983 829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:18.926465 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 829
I20260620 21:58:18.926568 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:58:18.926860 17679 external_mini_cluster.cc:1428] Setting key d646216c7a4de31eed1352253a46f560
Jun 20 21:58:18 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:18.934818 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:18.921116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:56295 (local address 127.17.67.254:44089)
0620 21:58:18.921269 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:18.921275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:18.922295 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 21:58:18.923624 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:18.923627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:18.923676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:18.923758 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:18.925027 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.925660 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.927082 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.927365 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.927486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:18.930475 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:18.930480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:18.930484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:18.930516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:18.932363 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.932926 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.932932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.932935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.933038 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:18.933405 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:18.933411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:18.933415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:18.933717 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:18.933837 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:18.934424 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 21:58:18.934573 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:58:18.935633 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:18.921689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:56295)
0620 21:58:18.922143 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:18.922187 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:18.923423 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:18.923918 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:18.923924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:18.924292 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:18.924824 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:18.924845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.925807 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.925810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:18.926946 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:18.926961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:18.927543 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:18.927547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:18.927758 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:18.928606 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:18.928626 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:18.930299 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:18.932487 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:18.932493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:18.932506 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:18.932772 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:18.933146 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:18.933149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:18.933150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:18.933293 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:18.933875 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:18.933881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:18.934145 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 21:58:18.934331 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 21:58:18.934605 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:58:18.936975 958 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:18.937258 958 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:18.937894 958 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:18.939379 633 ts_manager.cc:194] Registered new tserver with Master: 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
I20260620 21:58:18.940115 633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:56295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:58:18.956919 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:19.067376 963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:19.067633 963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:19.067713 963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:19.071710 963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:19.071769 963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:19.071869 963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:58:19.077173 963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:19.078265 963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:19.079311 963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:19.089541 969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:19.089529 971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:19.089488 968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:19.089948 963 server_base.cc:1034] running on GCE node
I20260620 21:58:19.090505 963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:19.091189 963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:19.092388 963 hybrid_clock.cc:648] HybridClock initialized: now 1781992699092370 us; error 33 us; skew 500 ppm
Jun 20 21:58:19 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:19.096804 963 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:58:19.098122 963 webserver.cc:460] Webserver started at http://127.17.67.195:45849/ using document root <none> and password file <none>
I20260620 21:58:19.098912 963 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:19.099010 963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:19.099288 963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:58:19.102005 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "482336e07ccc42dd9ad4def8289c5a9f"
format_stamp: "Formatted at 2026-06-20 21:58:19 on dist-test-slave-cx8d"
server_key: "59c95267f9b04237152326e9617f75ab"
server_key_iv: "4668da90829a5cfa8c01ff3737cbbebc"
server_key_version: "encryptionkey@0"
I20260620 21:58:19.102542 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "482336e07ccc42dd9ad4def8289c5a9f"
format_stamp: "Formatted at 2026-06-20 21:58:19 on dist-test-slave-cx8d"
server_key: "59c95267f9b04237152326e9617f75ab"
server_key_iv: "4668da90829a5cfa8c01ff3737cbbebc"
server_key_version: "encryptionkey@0"
I20260620 21:58:19.106454 963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:58:19.108971 978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:19.110312 963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:58:19.110473 963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "482336e07ccc42dd9ad4def8289c5a9f"
format_stamp: "Formatted at 2026-06-20 21:58:19 on dist-test-slave-cx8d"
server_key: "59c95267f9b04237152326e9617f75ab"
server_key_iv: "4668da90829a5cfa8c01ff3737cbbebc"
server_key_version: "encryptionkey@0"
I20260620 21:58:19.110590 963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:19.127163 963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:19.130366 963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:19.130579 963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:19.131181 963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:19.131999 963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:19.132071 963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:19.132140 963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:19.132193 963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:19.141026 963 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:44049
I20260620 21:58:19.141089 1091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:44049 every 8 connection(s)
I20260620 21:58:19.141990 963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:19.143404 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 963
I20260620 21:58:19.143570 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:58:19.143862 17679 external_mini_cluster.cc:1428] Setting key 73e3784dd39a681d3f090cc34b555f81
Jun 20 21:58:19 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:19.155656 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.144009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:45307 (local address 127.17.67.254:44089)
0620 21:58:19.144163 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.144168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.145081 (+ 913us) server_negotiation.cc:407] Connection header received
0620 21:58:19.146499 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.146501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.146545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.146611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:19.147749 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.148303 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.149456 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.149749 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.149838 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.152270 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:19.152274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:19.152276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:19.152300 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:19.153836 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:19.154329 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:19.154331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:19.154333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:19.154406 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:19.154693 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:19.154695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:19.154696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:19.154900 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:19.154977 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.155440 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.155534 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:58:19.156536 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.144449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:45307)
0620 21:58:19.144921 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.144974 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.146309 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.146743 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.146749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.147101 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:19.147600 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.147613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.148426 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.148428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.149342 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.149351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.149883 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.149886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.150025 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.150600 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:19.150616 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:19.152113 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:19.153974 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:19.153978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:19.153990 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:19.154245 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:19.154483 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:19.154485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:19.154486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:19.154616 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:19.154987 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:19.154992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:19.155211 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.155395 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.155634 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":138,"threads_started":1}
I20260620 21:58:19.157786 1092 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:19.158036 1092 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:19.158478 1092 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:19.159426 633 ts_manager.cc:194] Registered new tserver with Master: 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049)
I20260620 21:58:19.159931 633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:45307
I20260620 21:58:19.169668 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:58:19.178761 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.171652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.17.67.254:44089)
0620 21:58:19.171812 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.171818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.172077 (+ 259us) server_negotiation.cc:407] Connection header received
0620 21:58:19.172166 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.172168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.172211 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.172280 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:19.172962 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.173464 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.174381 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.174591 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.174669 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.176003 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:19.176006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:19.176008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:19.176029 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:19.177276 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:19.177718 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:19.177720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:19.177721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:19.177791 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:19.178018 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:19.178020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:19.178021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:19.178218 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:19.178317 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.178484 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.178567 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:58:19.181506 633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:19.184535 633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:19.198643 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.193173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.17.67.193:42239)
0620 21:58:19.193489 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.193506 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.194060 (+ 554us) server_negotiation.cc:407] Connection header received
0620 21:58:19.194147 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.194154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.194379 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.194545 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:19.194880 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.195782 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.197104 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.197969 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.198089 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.198237 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.198394 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.198492 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260620 21:58:19.198638 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.192988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:42239 (local address 127.0.0.1:40720)
0620 21:58:19.193675 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.193777 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.193914 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.194629 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.194634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.194655 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:19.194764 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.194769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.195913 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.195919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.196967 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.196978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.198087 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.198090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.198192 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.198227 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.198364 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.198485 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":98,"threads_started":1}
I20260620 21:58:19.199517 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.193171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44049 (local address 127.0.0.1:40454)
0620 21:58:19.193835 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.193848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.194201 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.195269 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.195273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.195290 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:19.195410 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.195417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.196904 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.196907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.198075 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.198087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.199126 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.199129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.199247 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.199269 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.199315 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.199380 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":42,"thread_start_us":123,"threads_started":1}
I20260620 21:58:19.199831 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.193748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39975 (local address 127.0.0.1:44842)
0620 21:58:19.194393 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.194415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.194497 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.195126 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.195129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.195148 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:19.195237 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.195243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.196584 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.196587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.197800 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.197811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.199442 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.199446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.199521 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.199551 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.199626 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.199698 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":23,"thread_start_us":90,"threads_started":1}
I20260620 21:58:19.200352 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.194286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.17.67.194:39975)
0620 21:58:19.194680 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.194694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.194719 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:58:19.194794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.194799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.194981 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.195125 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:19.195332 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.196446 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.197906 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.199126 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.199242 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.199387 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.200094 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.200194 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":101,"threads_started":1}
I20260620 21:58:19.200942 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.194158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.17.67.195:44049)
0620 21:58:19.194863 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.194873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.194891 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:58:19.194953 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.194959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.195119 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.195556 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:19.195695 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.196762 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.198211 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.198983 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.199781 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.200678 (+ 897us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.200724 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.200810 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":146,"threads_started":1}
I20260620 21:58:19.201889 758 tablet_service.cc:1467] Processing CreateTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 (DEFAULT_TABLE table=test-table [id=09cab11460c2484291c443ee55a73b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:19.202848 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3f42ca3cd44a4dbe68af7651723023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:19.202939 892 tablet_service.cc:1467] Processing CreateTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 (DEFAULT_TABLE table=test-table [id=09cab11460c2484291c443ee55a73b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:19.203471 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 (DEFAULT_TABLE table=test-table [id=09cab11460c2484291c443ee55a73b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:19.203955 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3f42ca3cd44a4dbe68af7651723023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:19.204427 1026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3f42ca3cd44a4dbe68af7651723023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:19.208659 1109 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Bootstrap starting.
I20260620 21:58:19.210937 1109 tablet_bootstrap.cc:654] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:19.211512 1110 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Bootstrap starting.
I20260620 21:58:19.212291 1109 log.cc:824] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:19.213909 1110 tablet_bootstrap.cc:654] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:19.213887 1111 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Bootstrap starting.
I20260620 21:58:19.214370 1109 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: No bootstrap required, opened a new log
I20260620 21:58:19.214561 1109 ts_tablet_manager.cc:1397] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 21:58:19.214740 1110 log.cc:824] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:19.215963 1111 tablet_bootstrap.cc:654] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:19.216627 1110 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: No bootstrap required, opened a new log
I20260620 21:58:19.216818 1110 ts_tablet_manager.cc:1397] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:58:19.217147 1111 log.cc:824] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:19.218552 1109 raft_consensus.cc:348] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.218817 1109 raft_consensus.cc:374] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:19.218876 1109 raft_consensus.cc:729] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b907c73706b14b8180da1da77aaa181b, State: Initialized, Role: FOLLOWER
I20260620 21:58:19.219107 1111 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: No bootstrap required, opened a new log
I20260620 21:58:19.219265 1111 ts_tablet_manager.cc:1397] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:58:19.219344 1109 consensus_queue.cc:260] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.220036 824 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
I20260620 21:58:19.220294 1109 ts_tablet_manager.cc:1428] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:58:19.220618 1110 raft_consensus.cc:348] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.220919 1110 raft_consensus.cc:374] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:19.220989 1110 raft_consensus.cc:729] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2642ef2c427d471c9bd20ea7a8b6026e, State: Initialized, Role: FOLLOWER
I20260620 21:58:19.221405 1110 consensus_queue.cc:260] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.222028 958 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
I20260620 21:58:19.222361 1110 ts_tablet_manager.cc:1428] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:58:19.223831 1111 raft_consensus.cc:348] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.224133 1111 raft_consensus.cc:374] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:19.224197 1111 raft_consensus.cc:729] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482336e07ccc42dd9ad4def8289c5a9f, State: Initialized, Role: FOLLOWER
I20260620 21:58:19.224622 1111 consensus_queue.cc:260] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.225625 1111 ts_tablet_manager.cc:1428] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:58:19.225785 1092 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
I20260620 21:58:19.378585 1116 raft_consensus.cc:482] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:19.378820 1116 raft_consensus.cc:504] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.380059 1116 leader_election.cc:288] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239), 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049)
I20260620 21:58:19.384505 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.380472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:54607 (local address 127.17.67.195:44049)
0620 21:58:19.380590 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.380595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.380610 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:19.380676 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.380679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.380726 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.380811 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:19.381136 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.381917 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.383256 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.383924 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.384052 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.384117 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.384339 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.384403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:58:19.384526 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.380346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44049 (local address 127.17.67.194:54607)
0620 21:58:19.380474 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.380492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.380590 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.380823 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.380826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.380843 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:19.380969 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.380974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.382042 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.382045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.383127 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.383135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.384061 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.384064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.384202 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.384228 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.384316 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.384389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 21:58:19.385313 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.380353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:42239 (local address 127.17.67.194:52033)
0620 21:58:19.380940 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.380952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:19.381083 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:19.381378 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:19.381381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:19.381396 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:19.381512 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.381517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.382885 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.382891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.384056 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:19.384065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.384997 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.385000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:19.385069 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.385086 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:58:19.385149 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:58:19.385196 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":165,"thread_start_us":165,"threads_started":1}
I20260620 21:58:19.385281 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3f42ca3cd44a4dbe68af7651723023" candidate_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482336e07ccc42dd9ad4def8289c5a9f" is_pre_election: true
I20260620 21:58:19.385376 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.380421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:52033 (local address 127.17.67.193:42239)
0620 21:58:19.380564 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.380570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.381053 (+ 483us) server_negotiation.cc:407] Connection header received
0620 21:58:19.381200 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.381203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.381259 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.381368 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:19.381934 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.382723 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.384191 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.384826 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.384968 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.385022 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.385186 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.385257 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:58:19.385671 1046 raft_consensus.cc:2454] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2642ef2c427d471c9bd20ea7a8b6026e in term 0.
I20260620 21:58:19.386102 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3f42ca3cd44a4dbe68af7651723023" candidate_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b907c73706b14b8180da1da77aaa181b" is_pre_election: true
I20260620 21:58:19.386157 848 leader_election.cc:302] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f; no voters:
I20260620 21:58:19.386384 778 raft_consensus.cc:2454] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2642ef2c427d471c9bd20ea7a8b6026e in term 0.
I20260620 21:58:19.386439 1116 raft_consensus.cc:2781] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:19.386554 1116 raft_consensus.cc:482] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:19.386610 1116 raft_consensus.cc:3037] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:19.387661 1116 raft_consensus.cc:504] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:19.387988 1116 leader_election.cc:288] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [CANDIDATE]: Term 1 election: Requested vote from peers b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239), 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049)
I20260620 21:58:19.388319 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3f42ca3cd44a4dbe68af7651723023" candidate_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482336e07ccc42dd9ad4def8289c5a9f"
I20260620 21:58:19.388342 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3f42ca3cd44a4dbe68af7651723023" candidate_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b907c73706b14b8180da1da77aaa181b"
I20260620 21:58:19.388423 1046 raft_consensus.cc:3037] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:19.388505 778 raft_consensus.cc:3037] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:19.389761 1046 raft_consensus.cc:2454] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2642ef2c427d471c9bd20ea7a8b6026e in term 1.
I20260620 21:58:19.389866 778 raft_consensus.cc:2454] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2642ef2c427d471c9bd20ea7a8b6026e in term 1.
I20260620 21:58:19.390136 848 leader_election.cc:302] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f; no voters:
I20260620 21:58:19.390354 1116 raft_consensus.cc:2781] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:19.390585 1116 raft_consensus.cc:686] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 LEADER]: Becoming Leader. State: Replica: 2642ef2c427d471c9bd20ea7a8b6026e, State: Running, Role: LEADER
I20260620 21:58:19.390931 1116 consensus_queue.cc:237] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
W20260620 21:58:19.393062 1093 tablet.cc:2369] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:19.394306 633 catalog_manager.cc:5495] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194). New cstate: current_term: 1 leader_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } health_report { overall_health: HEALTHY } } }
W20260620 21:58:19.420399 959 tablet.cc:2369] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:19.436103 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.432498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.17.67.194:39975)
0620 21:58:19.432632 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:19.432637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:19.432652 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:19.432778 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:19.432781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:19.432837 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:19.432984 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:19.433221 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.433806 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.434681 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:19.434896 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:19.435007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:19.435309 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:19.435729 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:19.435862 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:19.435902 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 21:58:19.435960 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:58:19.442102 1046 raft_consensus.cc:1261] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Refusing update from remote peer 2642ef2c427d471c9bd20ea7a8b6026e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:19.442102 778 raft_consensus.cc:1261] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Refusing update from remote peer 2642ef2c427d471c9bd20ea7a8b6026e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:19.442919 1119 consensus_queue.cc:1035] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [LEADER]: Connected to new peer: Peer: permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:19.443123 1116 consensus_queue.cc:1035] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:58:19.447889 825 tablet.cc:2369] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:19.451800 1123 mvcc.cc:205] Tried to move back new op lower bound from 7299042096906948608 to 7299042096708214784. Current Snapshot: MvccSnapshot[applied={T|T < 7299042096906948608}]
I20260620 21:58:19.461803 633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:19.462020 633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:19.464391 633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Sending DeleteTablet for 3 replicas of tablet ec3f42ca3cd44a4dbe68af7651723023
I20260620 21:58:19.465169 758 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:19 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:40720
I20260620 21:58:19.465250 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:19 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:40454
I20260620 21:58:19.465250 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:19 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:44842
I20260620 21:58:19.465716 1133 tablet_replica.cc:330] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: stopping tablet replica
I20260620 21:58:19.466005 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 695
I20260620 21:58:19.466027 1132 tablet_replica.cc:330] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: stopping tablet replica
I20260620 21:58:19.466681 1134 tablet_replica.cc:330] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: stopping tablet replica
I20260620 21:58:19.466980 1134 raft_consensus.cc:2229] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:19.467187 1133 raft_consensus.cc:2229] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:19.467556 1133 raft_consensus.cc:2258] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:19.468025 1134 raft_consensus.cc:2258] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:19.469403 1134 ts_tablet_manager.cc:1905] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:19.471957 1134 ts_tablet_manager.cc:1918] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:19.472076 1134 log.cc:1196] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/ec3f42ca3cd44a4dbe68af7651723023
I20260620 21:58:19.472378 1134 ts_tablet_manager.cc:1939] T ec3f42ca3cd44a4dbe68af7651723023 P 482336e07ccc42dd9ad4def8289c5a9f: Deleting consensus metadata
W20260620 21:58:19.473188 619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:42239 (error 108)
I20260620 21:58:19.473420 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 829
I20260620 21:58:19.474007 620 catalog_manager.cc:4841] TS 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): tablet ec3f42ca3cd44a4dbe68af7651723023 (table test-table [id=09cab11460c2484291c443ee55a73b19]) successfully deleted
I20260620 21:58:19.475582 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.475265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:42239 (local address 127.0.0.1:40730)
0620 21:58:19.475430 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.475500 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 21:58:19.475971 619 catalog_manager.cc:4584] TS b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec3f42ca3cd44a4dbe68af7651723023: Network error: Client connection negotiation failed: client connection to 127.17.67.193:42239: connect: Connection refused (error 111)
I20260620 21:58:19.479728 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 963
I20260620 21:58:19.480499 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:19.479959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39975 (local address 127.0.0.1:44852)
0620 21:58:19.480315 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:19.480395 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.194:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
I20260620 21:58:19.485991 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 601
I20260620 21:58:39.493086 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--webserver_interface=127.17.67.254
--webserver_port=39005
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:58:39.606706 1136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:39.607051 1136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:39.607127 1136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:39.611445 1136 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:58:39.611531 1136 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:58:39.611560 1136 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:58:39.611593 1136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:58:39.611637 1136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:39.611681 1136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:39.611719 1136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:58:39.611755 1136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:58:39.617175 1136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:39.618347 1136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:39.619406 1136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:39.632721 1145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.632740 1143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.632797 1142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:39.632802 1136 server_base.cc:1034] running on GCE node
I20260620 21:58:39.633811 1136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:39.634989 1136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:39.636197 1136 hybrid_clock.cc:648] HybridClock initialized: now 1781992719636179 us; error 33 us; skew 500 ppm
Jun 20 21:58:39 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:39.639129 1136 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:58:39.640234 1136 webserver.cc:460] Webserver started at http://127.17.67.254:39005/ using document root <none> and password file <none>
I20260620 21:58:39.640866 1136 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:39.640985 1136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:39.645911 1136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:39.648422 1152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.649881 1136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:39.650029 1136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "abab11eeb87149358d20d72b2ec37aea"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "f0ccb1eedae0f2bfcbe83abc02c127f4"
server_key_iv: "a21eb84b3a7762be08677f5f844bc6b4"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.650655 1136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:39.672830 1136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:39.676368 1136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:39.676622 1136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:39.684698 1136 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44089
I20260620 21:58:39.685161 1204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44089 every 8 connection(s)
I20260620 21:58:39.685995 1136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:58:39.690477 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1136
I20260620 21:58:39.690550 1205 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:58:39.690976 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:42239
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=36699
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:39.692595 1205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap starting.
I20260620 21:58:39.711035 1205 log.cc:824] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:39.719630 1205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:39.720038 1205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap complete.
I20260620 21:58:39.724112 1205 raft_consensus.cc:348] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:39.724543 1205 raft_consensus.cc:729] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Initialized, Role: FOLLOWER
I20260620 21:58:39.725023 1205 consensus_queue.cc:260] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:39.725167 1205 raft_consensus.cc:388] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:58:39.725261 1205 raft_consensus.cc:482] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:58:39.725379 1205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:58:39.727319 1205 raft_consensus.cc:504] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:39.727660 1205 leader_election.cc:302] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abab11eeb87149358d20d72b2ec37aea; no voters:
I20260620 21:58:39.728017 1205 leader_election.cc:288] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:58:39.728098 1209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:58:39.728307 1209 raft_consensus.cc:686] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 LEADER]: Becoming Leader. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Running, Role: LEADER
I20260620 21:58:39.728660 1209 consensus_queue.cc:237] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:58:39.729322 1205 sys_catalog.cc:564] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:58:39.729636 1209 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader abab11eeb87149358d20d72b2ec37aea. Latest consensus state: current_term: 2 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:58:39.729758 1209 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:39.730154 1210 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:58:39.730333 1210 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:58:39.730623 1213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:58:39.736140 1213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:58:39.736948 1213 catalog_manager.cc:1259] Loaded cluster ID: 1b02d91c4cd54ffd959202b75ae04f6e
I20260620 21:58:39.737015 1213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:58:39.739461 1213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:58:39.740170 1213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Loaded TSK: 0
I20260620 21:58:39.759130 1213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Generated new TSK 1
I20260620 21:58:39.759466 1213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:58:39.822394 1207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:39.822681 1207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:39.822732 1207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:39.827494 1207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:39.827564 1207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:39.827665 1207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:58:39.833963 1207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:42239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=36699
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:39.835127 1207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:39.836256 1207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:39.847998 1232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.848668 1233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:39.853806 1235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:39.855396 1207 server_base.cc:1034] running on GCE node
I20260620 21:58:39.855914 1207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:39.856662 1207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:39.857867 1207 hybrid_clock.cc:648] HybridClock initialized: now 1781992719857837 us; error 56 us; skew 500 ppm
Jun 20 21:58:39 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:39.860993 1207 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:58:39.862162 1207 webserver.cc:460] Webserver started at http://127.17.67.193:36699/ using document root <none> and password file <none>
I20260620 21:58:39.862872 1207 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:39.862991 1207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:39.867717 1207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:58:39.871256 1242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:39.873121 1207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:58:39.873292 1207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "b907c73706b14b8180da1da77aaa181b"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "64ac235e269334a1f4be282e9a3475de"
server_key_iv: "460272550ff3db30110d0066a68796c1"
server_key_version: "encryptionkey@0"
I20260620 21:58:39.874156 1207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:39.924408 1207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:39.927918 1207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:39.928148 1207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:39.928845 1207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:39.930125 1249 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:58:39.934427 1207 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:58:39.934536 1207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 21:58:39.934626 1207 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:58:39.936450 1207 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:58:39.936528 1207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:58:39.936817 1249 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Bootstrap starting.
I20260620 21:58:39.948719 1207 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:42239
I20260620 21:58:39.948835 1356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:42239 every 8 connection(s)
I20260620 21:58:39.949910 1207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:58:39.955492 1249 log.cc:824] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:39.955543 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1207
I20260620 21:58:39.956173 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:39975
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=40069
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:58:39 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:39.963567 1249 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:39.964054 1249 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Bootstrap complete.
I20260620 21:58:39.965301 1249 ts_tablet_manager.cc:1397] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20260620 21:58:39.968128 1249 raft_consensus.cc:348] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:39.968677 1249 raft_consensus.cc:729] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b907c73706b14b8180da1da77aaa181b, State: Initialized, Role: FOLLOWER
I20260620 21:58:39.969242 1249 consensus_queue.cc:260] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:39.970235 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.952183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:51251)
0620 21:58:39.952825 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:39.952902 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:39.954060 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:39.955425 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:39.955437 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:39.955921 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:39.956543 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.956567 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.958459 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.958470 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.959391 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.959401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.960205 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.960210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.960418 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.961111 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:39.961138 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:39.963136 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:39.966255 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.966261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.966275 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.966613 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.967078 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.967081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.967083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.967297 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.968132 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:39.968140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:39.968445 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 21:58:39.968820 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 21:58:39.969160 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":158,"threads_started":1}
I20260620 21:58:39.970449 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.952189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:51251 (local address 127.17.67.254:44089)
0620 21:58:39.952644 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:39.952666 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:39.953054 (+ 388us) server_negotiation.cc:407] Connection header received
0620 21:58:39.954531 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:39.954544 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:39.954976 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:39.955339 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:39.956748 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.958281 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.959528 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.960031 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.960228 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.963390 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:39.963399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:39.963422 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:39.963469 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:39.966106 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.966750 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.966760 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.966769 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.966959 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.967417 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.967423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.967427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.967986 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:39.968151 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:39.968834 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 21:58:39.969208 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260620 21:58:39.970314 1249 ts_tablet_manager.cc:1428] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260620 21:58:39.972957 1357 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:39.973203 1357 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:39.973793 1357 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:39.976315 1169 ts_manager.cc:194] Registered new tserver with Master: b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239)
I20260620 21:58:39.979890 1169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:51251
I20260620 21:58:39.982785 1357 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
Jun 20 21:58:39 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.193@KRBTEST.COM
I20260620 21:58:39.990382 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.979394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.17.67.193:42239)
0620 21:58:39.979887 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:39.979899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:39.980292 (+ 393us) server_negotiation.cc:407] Connection header received
0620 21:58:39.980479 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:39.980487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:39.980710 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:39.980851 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:39.981500 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.982264 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.983337 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.983794 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.983920 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.986726 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:39.986735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:39.986745 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:39.986767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:39.988364 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.988938 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.988942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.988943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.989025 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:39.989313 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:39.989315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:39.989316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:39.989545 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:39.989734 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:39.990107 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 21:58:39.990254 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":111,"threads_started":1}
I20260620 21:58:39.990449 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:39.979277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:42239 (local address 127.0.0.1:42548)
0620 21:58:39.980134 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:39.980178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:39.980318 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:39.980893 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:39.980900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:39.981250 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:39.981355 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.981362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.982382 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.982388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.983192 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:39.983211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:39.983936 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:39.983942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:39.984064 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:39.984740 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:39.984758 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:39.986579 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:39.988479 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.988488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.988494 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.988822 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.989118 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:39.989121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:39.989123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:39.989223 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:39.989724 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:39.989730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:39.989871 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 21:58:39.990032 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 21:58:39.990234 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":97,"threads_started":1}
I20260620 21:58:39.991618 1291 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:19 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:42548
I20260620 21:58:39.992473 1366 tablet_replica.cc:330] stopping tablet replica
I20260620 21:58:39.992792 1366 raft_consensus.cc:2229] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:39.993043 1366 raft_consensus.cc:2258] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:39.994024 1366 ts_tablet_manager.cc:1905] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:39.997664 1366 ts_tablet_manager.cc:1918] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:39.997785 1366 log.cc:1196] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal/wals/ec3f42ca3cd44a4dbe68af7651723023
I20260620 21:58:39.998093 1366 ts_tablet_manager.cc:1939] T ec3f42ca3cd44a4dbe68af7651723023 P b907c73706b14b8180da1da77aaa181b: Deleting consensus metadata
I20260620 21:58:39.999704 1155 catalog_manager.cc:4841] TS b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239): tablet ec3f42ca3cd44a4dbe68af7651723023 (table test-table [id=09cab11460c2484291c443ee55a73b19]) successfully deleted
W20260620 21:58:40.084161 1362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:40.084496 1362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:40.084576 1362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:40.088547 1362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:40.088621 1362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:40.088680 1362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:58:40.093868 1362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:39975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=40069
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:40.094978 1362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:40.095986 1362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:40.105500 1371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:40.105547 1372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.105799 1362 server_base.cc:1034] running on GCE node
W20260620 21:58:40.105544 1374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.106348 1362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:40.107045 1362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:40.108263 1362 hybrid_clock.cc:648] HybridClock initialized: now 1781992720108225 us; error 49 us; skew 500 ppm
Jun 20 21:58:40 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:40.111492 1362 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:58:40.112434 1362 webserver.cc:460] Webserver started at http://127.17.67.194:40069/ using document root <none> and password file <none>
I20260620 21:58:40.113057 1362 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:40.113181 1362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:40.117874 1362 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:58:40.120332 1381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.121687 1362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:40.121855 1362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "fc6c0b465067c934c739780f106cdf4a"
server_key_iv: "cfdd82c91d1ee704effbe3e00741af25"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.122469 1362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:40.144415 1362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:40.148353 1362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:40.148622 1362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:40.149308 1362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:40.150519 1388 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:58:40.155077 1362 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:58:40.155167 1362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 21:58:40.155260 1362 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:58:40.157009 1362 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:58:40.157081 1362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:58:40.157415 1388 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Bootstrap starting.
I20260620 21:58:40.167603 1362 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:39975
I20260620 21:58:40.167677 1495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:39975 every 8 connection(s)
I20260620 21:58:40.168618 1362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:58:40.176376 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1362
I20260620 21:58:40.176935 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:44049
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=45849
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:58:40.177889 1388 log.cc:824] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:58:40 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:40.185057 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.170795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:44093 (local address 127.17.67.254:44089)
0620 21:58:40.171030 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.171039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.171616 (+ 577us) server_negotiation.cc:407] Connection header received
0620 21:58:40.172552 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.172559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.172636 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.172761 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.174007 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.174934 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.176158 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.176519 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.176640 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.180095 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.180102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.180109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.180170 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.182300 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.182993 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.183000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.183003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.183117 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.183533 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.183540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.183543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.183866 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.183995 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.184591 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.184785 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:58:40.185791 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.170919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:44093)
0620 21:58:40.171442 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.171495 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.172329 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.172903 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.172909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.173287 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:40.173844 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.173859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.175090 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.175093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.176002 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.176022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.176655 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.176660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.176893 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.177747 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:40.177774 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:40.179925 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:40.182440 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.182446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.182458 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.182871 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.183237 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.183240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.183242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.183419 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.183994 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:40.184000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:40.184301 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.184503 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.184829 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260620 21:58:40.186664 1388 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:58:40.187050 1496 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:40.187093 1388 tablet_bootstrap.cc:492] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Bootstrap complete.
I20260620 21:58:40.187340 1496 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:40.188021 1496 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:40.188297 1388 ts_tablet_manager.cc:1397] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20260620 21:58:40.190496 1169 ts_manager.cc:194] Registered new tserver with Master: 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
I20260620 21:58:40.190915 1388 raft_consensus.cc:348] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.191433 1388 raft_consensus.cc:729] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2642ef2c427d471c9bd20ea7a8b6026e, State: Initialized, Role: FOLLOWER
I20260620 21:58:40.191840 1169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:44093
I20260620 21:58:40.191911 1388 consensus_queue.cc:260] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.192930 1388 ts_tablet_manager.cc:1428] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:58:40.194344 1496 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
I20260620 21:58:40.197237 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.191496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39975 (local address 127.0.0.1:41368)
0620 21:58:40.191820 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.191838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.191932 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.193778 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.193781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.193804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:40.193914 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.193921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.195127 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.195133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.195951 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.195960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.196723 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.196727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.196827 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.196889 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.197007 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.197100 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260620 21:58:40.197237 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.191764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.17.67.194:39975)
0620 21:58:40.193297 (+ 1533us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.193308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.193330 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:58:40.193393 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.193398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.193607 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.193778 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:40.194041 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.195021 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.196087 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.196599 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.196731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.196863 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.197040 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.197120 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":121,"threads_started":1}
I20260620 21:58:40.198237 1430 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3f42ca3cd44a4dbe68af7651723023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:19 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:41368
I20260620 21:58:40.199106 1503 tablet_replica.cc:330] stopping tablet replica
I20260620 21:58:40.199438 1503 raft_consensus.cc:2229] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:40.199712 1503 raft_consensus.cc:2258] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:40.200842 1503 ts_tablet_manager.cc:1905] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:40.205842 1503 ts_tablet_manager.cc:1918] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:40.205950 1503 log.cc:1196] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/ec3f42ca3cd44a4dbe68af7651723023
I20260620 21:58:40.206239 1503 ts_tablet_manager.cc:1939] T ec3f42ca3cd44a4dbe68af7651723023 P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting consensus metadata
I20260620 21:58:40.208140 1156 catalog_manager.cc:4841] TS 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): tablet ec3f42ca3cd44a4dbe68af7651723023 (table test-table [id=09cab11460c2484291c443ee55a73b19]) successfully deleted
W20260620 21:58:40.302038 1500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:58:40.302307 1500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:58:40.302353 1500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:58:40.306550 1500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:58:40.306613 1500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:58:40.306680 1500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:58:40.312145 1500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:44049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=45849
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:58:40.313287 1500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:58:40.314419 1500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:58:40.324429 1508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:40.324409 1511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:58:40.324447 1509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:58:40.324654 1500 server_base.cc:1034] running on GCE node
I20260620 21:58:40.325241 1500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:58:40.326033 1500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:58:40.327240 1500 hybrid_clock.cc:648] HybridClock initialized: now 1781992720327220 us; error 38 us; skew 500 ppm
Jun 20 21:58:40 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:58:40.330659 1500 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:58:40.331578 1500 webserver.cc:460] Webserver started at http://127.17.67.195:45849/ using document root <none> and password file <none>
I20260620 21:58:40.332139 1500 fs_manager.cc:359] Metadata directory not provided
I20260620 21:58:40.332222 1500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:58:40.336515 1500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:58:40.338825 1518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.340240 1500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:58:40.340341 1500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "482336e07ccc42dd9ad4def8289c5a9f"
format_stamp: "Formatted at 2026-06-20 21:58:19 on dist-test-slave-cx8d"
server_key: "59c95267f9b04237152326e9617f75ab"
server_key_iv: "4668da90829a5cfa8c01ff3737cbbebc"
server_key_version: "encryptionkey@0"
I20260620 21:58:40.340965 1500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:58:40.356769 1500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:58:40.360270 1500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:58:40.360488 1500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:58:40.361121 1500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:58:40.362035 1500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:58:40.362082 1500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.362121 1500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:58:40.362138 1500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:58:40.371789 1500 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:44049
I20260620 21:58:40.371811 1631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:44049 every 8 connection(s)
I20260620 21:58:40.372818 1500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:58:40.375315 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1500
Jun 20 21:58:40 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:58:40.385951 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.376036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.17.67.254:44089)
0620 21:58:40.376270 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.376274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.376576 (+ 302us) server_negotiation.cc:407] Connection header received
0620 21:58:40.376760 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.376763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.376814 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.376906 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.377893 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.378745 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.379879 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.380265 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.380385 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.382059 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.382063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.382065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.382091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.383975 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.384466 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.384470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.384472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.384574 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.384848 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.384852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.384853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.385086 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.385322 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.385516 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.385707 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":60,"threads_started":1}
I20260620 21:58:40.389547 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.374817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:50539 (local address 127.17.67.254:44089)
0620 21:58:40.374972 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.374977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.376000 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 21:58:40.377937 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.378080 (+ 143us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.378143 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.378247 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.379688 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.380576 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.382277 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.382652 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.382791 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.385525 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.385529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.385533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.385608 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.387551 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.388099 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.388102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.388103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.388178 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.388504 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.388507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.388508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.388730 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.388819 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.389299 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.389425 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260620 21:58:40.390456 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.375272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:50539)
0620 21:58:40.375837 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.375900 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.377696 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.378417 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.378424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.378779 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:58:40.379453 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.379472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.380693 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.380699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.382161 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.382171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.382811 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.382814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.382960 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.383572 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:58:40.383589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:58:40.385355 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:58:40.387661 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.387666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.387678 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.388027 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.388264 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:58:40.388267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:58:40.388268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:58:40.388399 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:58:40.388824 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:58:40.388829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:58:40.389057 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.389221 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.389486 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":152,"threads_started":1}
I20260620 21:58:40.391607 1632 heartbeater.cc:347] Connected to a master server at 127.17.67.254:44089
I20260620 21:58:40.391892 1632 heartbeater.cc:464] Registering TS with master...
I20260620 21:58:40.392455 1632 heartbeater.cc:511] Master 127.17.67.254:44089 requested a full tablet report, sending...
I20260620 21:58:40.393484 1169 ts_manager.cc:194] Registered new tserver with Master: 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049)
I20260620 21:58:40.394052 1169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:50539
I20260620 21:58:40.399907 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:58:40.404641 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.400695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.17.67.254:44089)
0620 21:58:40.400848 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.400853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.400963 (+ 110us) server_negotiation.cc:407] Connection header received
0620 21:58:40.401190 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.401193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.401295 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.401416 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:40.401753 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.402328 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.403375 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.403607 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.403726 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.404073 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:40.404220 (+ 147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 21:58:40.404494 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49336: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 21:58:40.404733 1359 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49336: token expired
W20260620 21:58:40.405035 1098 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.67.254:44089)
I20260620 21:58:40.412842 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.405480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.17.67.254:44089)
0620 21:58:40.405644 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.405648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.405663 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:40.405706 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.405710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.405754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.405837 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:58:40.406451 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.406937 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.407764 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.407973 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.408084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.409433 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:58:40.409437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:58:40.409439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:58:40.409464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:58:40.411020 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.411507 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.411512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.411518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.411613 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:58:40.411927 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:58:40.411931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:58:40.411933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:58:40.412124 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:58:40.412288 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.412543 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.412669 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:58:40.416111 1169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:58:40.419726 1169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:58:40.428193 1430 tablet_service.cc:1467] Processing CreateTablet for tablet f1ea04a9212d48358c948c32efd0a75c (DEFAULT_TABLE table=test-table [id=32ecf28c4ef0431288c759362f6276f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:40.428285 1291 tablet_service.cc:1467] Processing CreateTablet for tablet f1ea04a9212d48358c948c32efd0a75c (DEFAULT_TABLE table=test-table [id=32ecf28c4ef0431288c759362f6276f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:40.428882 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ea04a9212d48358c948c32efd0a75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:40.428906 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ea04a9212d48358c948c32efd0a75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:40.431666 1249 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Bootstrap starting.
I20260620 21:58:40.431680 1388 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Bootstrap starting.
I20260620 21:58:40.432534 1388 tablet_bootstrap.cc:654] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:40.432549 1249 tablet_bootstrap.cc:654] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:40.433807 1388 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: No bootstrap required, opened a new log
I20260620 21:58:40.433892 1388 ts_tablet_manager.cc:1397] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:58:40.434016 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.427441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44049 (local address 127.0.0.1:37594)
0620 21:58:40.427668 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.427683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.427755 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.428643 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.428647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.428665 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:40.428771 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.428777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.430719 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.430726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.431871 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.431882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.433654 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.433659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.433775 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.433799 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.433845 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.433917 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 21:58:40.434052 1249 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: No bootstrap required, opened a new log
I20260620 21:58:40.434131 1249 ts_tablet_manager.cc:1397] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 21:58:40.434592 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.427627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.17.67.195:44049)
0620 21:58:40.428039 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.428052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.428075 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:58:40.428154 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.428159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.428475 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.428665 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:40.428876 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.430582 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.432019 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.433467 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.434215 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.434354 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.434399 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.434471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":174,"threads_started":1}
I20260620 21:58:40.434584 1388 raft_consensus.cc:348] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.434584 1249 raft_consensus.cc:348] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.434691 1249 raft_consensus.cc:374] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:40.434691 1388 raft_consensus.cc:374] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:40.434732 1249 raft_consensus.cc:729] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b907c73706b14b8180da1da77aaa181b, State: Initialized, Role: FOLLOWER
I20260620 21:58:40.434743 1388 raft_consensus.cc:729] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2642ef2c427d471c9bd20ea7a8b6026e, State: Initialized, Role: FOLLOWER
I20260620 21:58:40.434861 1388 consensus_queue.cc:260] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.434861 1249 consensus_queue.cc:260] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.435113 1388 ts_tablet_manager.cc:1428] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:58:40.435115 1249 ts_tablet_manager.cc:1428] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:58:40.438037 1566 tablet_service.cc:1467] Processing CreateTablet for tablet f1ea04a9212d48358c948c32efd0a75c (DEFAULT_TABLE table=test-table [id=32ecf28c4ef0431288c759362f6276f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:58:40.438954 1566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ea04a9212d48358c948c32efd0a75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:58:40.441303 1363 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:58:40.441500 1363 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.442584 1363 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
I20260620 21:58:40.447608 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.443196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:37563 (local address 127.17.67.194:39975)
0620 21:58:40.443319 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.443324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.443591 (+ 267us) server_negotiation.cc:407] Connection header received
0620 21:58:40.443744 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.443748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.443796 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.443877 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:40.444165 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.444870 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.446500 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.446981 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.447121 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.447176 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.447395 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.447471 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:58:40.447649 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.442757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44049 (local address 127.17.67.193:41159)
0620 21:58:40.442899 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.442915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.443006 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.443755 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.443758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.443776 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:40.443894 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.443901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.444999 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.445002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.446487 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.446497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.447329 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.447332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.447423 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.447442 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.447482 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.447534 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":270}
I20260620 21:58:40.447651 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.443019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39975 (local address 127.17.67.193:37563)
0620 21:58:40.443496 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.443508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:58:40.443629 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:58:40.443899 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:58:40.443902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:58:40.443919 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:58:40.444051 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.444057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.444984 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.444988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.446367 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:58:40.446378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.447129 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.447133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:58:40.447238 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.447284 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:58:40.447382 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 21:58:40.447459 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":26,"thread_start_us":105,"threads_started":1}
I20260620 21:58:40.448287 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.443397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:41159 (local address 127.17.67.195:44049)
0620 21:58:40.443524 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.443528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.443543 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:40.443591 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.443593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.443636 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.443997 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:58:40.444089 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.444868 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.446641 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.447196 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.448014 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.448058 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.448101 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.448182 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:58:40.448388 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ea04a9212d48358c948c32efd0a75c" candidate_uuid: "b907c73706b14b8180da1da77aaa181b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" is_pre_election: true
I20260620 21:58:40.448695 1450 raft_consensus.cc:2454] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b907c73706b14b8180da1da77aaa181b in term 0.
I20260620 21:58:40.448747 1645 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Bootstrap starting.
I20260620 21:58:40.449180 1246 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2642ef2c427d471c9bd20ea7a8b6026e, b907c73706b14b8180da1da77aaa181b; no voters:
I20260620 21:58:40.449502 1363 raft_consensus.cc:2781] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:58:40.449635 1363 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:58:40.449702 1363 raft_consensus.cc:3037] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:40.448824 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ea04a9212d48358c948c32efd0a75c" candidate_uuid: "b907c73706b14b8180da1da77aaa181b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482336e07ccc42dd9ad4def8289c5a9f" is_pre_election: true
W20260620 21:58:40.450887 1246 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Illegal state: must be running to vote when last-logged opid is not known
I20260620 21:58:40.451007 1363 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
W20260620 21:58:40.451231 1358 tablet.cc:2369] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:58:40.451383 1363 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 election: Requested vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
I20260620 21:58:40.451658 1645 tablet_bootstrap.cc:654] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Neither blocks nor log segments found. Creating new log.
I20260620 21:58:40.451774 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ea04a9212d48358c948c32efd0a75c" candidate_uuid: "b907c73706b14b8180da1da77aaa181b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482336e07ccc42dd9ad4def8289c5a9f"
I20260620 21:58:40.451805 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ea04a9212d48358c948c32efd0a75c" candidate_uuid: "b907c73706b14b8180da1da77aaa181b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
I20260620 21:58:40.451946 1450 raft_consensus.cc:3037] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 0 FOLLOWER]: Advancing to term 1
W20260620 21:58:40.452291 1246 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Illegal state: must be running to vote when last-logged opid is not known
I20260620 21:58:40.452538 1645 log.cc:824] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:58:40.453217 1450 raft_consensus.cc:2454] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b907c73706b14b8180da1da77aaa181b in term 1.
I20260620 21:58:40.453500 1246 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2642ef2c427d471c9bd20ea7a8b6026e, b907c73706b14b8180da1da77aaa181b; no voters: 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:58:40.453711 1363 raft_consensus.cc:2781] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:58:40.453974 1363 raft_consensus.cc:686] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 LEADER]: Becoming Leader. State: Replica: b907c73706b14b8180da1da77aaa181b, State: Running, Role: LEADER
I20260620 21:58:40.454968 1645 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: No bootstrap required, opened a new log
I20260620 21:58:40.455242 1645 ts_tablet_manager.cc:1397] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:58:40.454231 1363 consensus_queue.cc:237] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.458038 1645 raft_consensus.cc:348] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.458292 1645 raft_consensus.cc:374] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:58:40.458353 1645 raft_consensus.cc:729] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482336e07ccc42dd9ad4def8289c5a9f, State: Initialized, Role: FOLLOWER
I20260620 21:58:40.458856 1645 consensus_queue.cc:260] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:58:40.458817 1168 catalog_manager.cc:5495] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b reported cstate change: term changed from 0 to 1, leader changed from <none> to b907c73706b14b8180da1da77aaa181b (127.17.67.193). New cstate: current_term: 1 leader_uuid: "b907c73706b14b8180da1da77aaa181b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } health_report { overall_health: UNKNOWN } } }
I20260620 21:58:40.459525 1632 heartbeater.cc:503] Master 127.17.67.254:44089 was elected leader, sending a full tablet report...
I20260620 21:58:40.459993 1645 ts_tablet_manager.cc:1428] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:58:40.476994 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.473305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.17.67.193:42239)
0620 21:58:40.473458 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:58:40.473465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:58:40.473480 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:58:40.473650 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:58:40.473654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:58:40.473723 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:58:40.473843 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:58:40.474083 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.474641 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.475491 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:58:40.475746 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:58:40.475852 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:58:40.476147 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:58:40.476603 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:58:40.476743 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:58:40.476781 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 21:58:40.476834 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:58:40.481235 1586 raft_consensus.cc:3037] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:58:40.481712 1450 raft_consensus.cc:1261] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Refusing update from remote peer b907c73706b14b8180da1da77aaa181b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:40.482446 1647 consensus_queue.cc:1035] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:40.482749 1586 raft_consensus.cc:1261] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Refusing update from remote peer b907c73706b14b8180da1da77aaa181b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:58:40.483414 1647 consensus_queue.cc:1035] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [LEADER]: Connected to new peer: Peer: permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:58:40.487591 1651 mvcc.cc:205] Tried to move back new op lower bound from 7299042183084425216 to 7299042182988349440. Current Snapshot: MvccSnapshot[applied={T|T < 7299042183084425216}]
I20260620 21:58:40.490200 1654 mvcc.cc:205] Tried to move back new op lower bound from 7299042183084425216 to 7299042182988349440. Current Snapshot: MvccSnapshot[applied={T|T < 7299042183084425216}]
I20260620 21:58:40.497231 1168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:40.497386 1168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:58:40.499866 1168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Sending DeleteTablet for 3 replicas of tablet f1ea04a9212d48358c948c32efd0a75c
I20260620 21:58:40.500535 1291 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ea04a9212d48358c948c32efd0a75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:40 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:42548
I20260620 21:58:40.500708 1430 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ea04a9212d48358c948c32efd0a75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:40 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:41368
I20260620 21:58:40.500744 1566 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ea04a9212d48358c948c32efd0a75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:58:40 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:37594
I20260620 21:58:40.500869 1660 tablet_replica.cc:330] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: stopping tablet replica
I20260620 21:58:40.500891 1503 tablet_replica.cc:330] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: stopping tablet replica
I20260620 21:58:40.500974 1660 raft_consensus.cc:2229] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:58:40.500990 1503 raft_consensus.cc:2229] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:40.501108 1503 raft_consensus.cc:2258] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:40.501272 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1207
I20260620 21:58:40.501283 1660 raft_consensus.cc:2258] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:40.502482 1503 ts_tablet_manager.cc:1905] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:40.503284 1661 tablet_replica.cc:330] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: stopping tablet replica
I20260620 21:58:40.503609 1661 raft_consensus.cc:2229] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:58:40.503949 1661 raft_consensus.cc:2258] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:58:40.504981 1503 ts_tablet_manager.cc:1918] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:40.505069 1503 log.cc:1196] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/f1ea04a9212d48358c948c32efd0a75c
I20260620 21:58:40.505267 1661 ts_tablet_manager.cc:1905] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:58:40.505328 1503 ts_tablet_manager.cc:1939] T f1ea04a9212d48358c948c32efd0a75c P 2642ef2c427d471c9bd20ea7a8b6026e: Deleting consensus metadata
I20260620 21:58:40.506126 1156 catalog_manager.cc:4841] TS 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): tablet f1ea04a9212d48358c948c32efd0a75c (table test-table [id=32ecf28c4ef0431288c759362f6276f2]) successfully deleted
I20260620 21:58:40.507645 1661 ts_tablet_manager.cc:1918] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:58:40.507768 1661 log.cc:1196] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/f1ea04a9212d48358c948c32efd0a75c
I20260620 21:58:40.508091 1661 ts_tablet_manager.cc:1939] T f1ea04a9212d48358c948c32efd0a75c P 482336e07ccc42dd9ad4def8289c5a9f: Deleting consensus metadata
I20260620 21:58:40.509081 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1362
W20260620 21:58:40.509124 1155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:42239 (error 108)
I20260620 21:58:40.510368 1156 catalog_manager.cc:4841] TS 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): tablet f1ea04a9212d48358c948c32efd0a75c (table test-table [id=32ecf28c4ef0431288c759362f6276f2]) successfully deleted
I20260620 21:58:40.514241 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:58:40.513926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:42239 (local address 127.0.0.1:42576)
0620 21:58:40.514085 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:58:40.514166 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 21:58:40.514420 1155 catalog_manager.cc:4584] TS b907c73706b14b8180da1da77aaa181b (127.17.67.193:42239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1ea04a9212d48358c948c32efd0a75c: Network error: Client connection negotiation failed: client connection to 127.17.67.193:42239: connect: Connection refused (error 111)
I20260620 21:58:40.515113 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1500
I20260620 21:58:40.521036 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1136
I20260620 21:59:00.529942 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--webserver_interface=127.17.67.254
--webserver_port=39005
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:00.645005 1663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:00.645282 1663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:00.645324 1663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:00.649395 1663 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:59:00.649456 1663 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:59:00.649471 1663 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:59:00.649485 1663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:00.649504 1663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:00.649583 1663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:00.649607 1663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:00.649637 1663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:00.654886 1663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:44089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:00.655977 1663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:00.657053 1663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:00.666040 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:00.666181 1669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:00.666141 1663 server_base.cc:1034] running on GCE node
W20260620 21:59:00.666239 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:00.666944 1663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:00.668087 1663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:00.669275 1663 hybrid_clock.cc:648] HybridClock initialized: now 1781992740669235 us; error 54 us; skew 500 ppm
Jun 20 21:59:00 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992740, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:00.672334 1663 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:00.673303 1663 webserver.cc:460] Webserver started at http://127.17.67.254:39005/ using document root <none> and password file <none>
I20260620 21:59:00.673969 1663 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:00.674042 1663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:00.678570 1663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:00.680855 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:00.682200 1663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:00.682351 1663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "abab11eeb87149358d20d72b2ec37aea"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "f0ccb1eedae0f2bfcbe83abc02c127f4"
server_key_iv: "a21eb84b3a7762be08677f5f844bc6b4"
server_key_version: "encryptionkey@0"
I20260620 21:59:00.682925 1663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:00.695044 1663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:00.698369 1663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:00.698704 1663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:00.705943 1663 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:44089
I20260620 21:59:00.706254 1730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:44089 every 8 connection(s)
I20260620 21:59:00.707109 1663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:00.711294 1731 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:59:00.712816 1731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap starting.
I20260620 21:59:00.716931 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1663
I20260620 21:59:00.717382 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:42239
--local_ip_for_outbound_sockets=127.17.67.193
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=36699
--webserver_interface=127.17.67.193
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:00.730103 1731 log.cc:824] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:00.741125 1731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:59:00.741509 1731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: Bootstrap complete.
I20260620 21:59:00.745463 1731 raft_consensus.cc:348] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:59:00.745956 1731 raft_consensus.cc:729] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Initialized, Role: FOLLOWER
I20260620 21:59:00.746500 1731 consensus_queue.cc:260] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:59:00.746649 1731 raft_consensus.cc:388] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:00.746738 1731 raft_consensus.cc:482] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:00.746852 1731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:59:00.748708 1731 raft_consensus.cc:504] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:59:00.749037 1731 leader_election.cc:302] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abab11eeb87149358d20d72b2ec37aea; no voters:
I20260620 21:59:00.749361 1731 leader_election.cc:288] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:59:00.749504 1735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:59:00.749782 1735 raft_consensus.cc:686] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [term 3 LEADER]: Becoming Leader. State: Replica: abab11eeb87149358d20d72b2ec37aea, State: Running, Role: LEADER
I20260620 21:59:00.750166 1735 consensus_queue.cc:237] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } }
I20260620 21:59:00.750593 1731 sys_catalog.cc:564] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:00.751521 1736 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:59:00.751660 1736 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:00.753062 1737 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader abab11eeb87149358d20d72b2ec37aea. Latest consensus state: current_term: 3 leader_uuid: "abab11eeb87149358d20d72b2ec37aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab11eeb87149358d20d72b2ec37aea" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 44089 } } }
I20260620 21:59:00.753667 1737 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:00.756381 1751 catalog_manager.cc:1259] Loaded cluster ID: 1b02d91c4cd54ffd959202b75ae04f6e
I20260620 21:59:00.756444 1751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: loading cluster ID for follower catalog manager: success
I20260620 21:59:00.758972 1751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P abab11eeb87149358d20d72b2ec37aea: acquiring CA information for follower catalog manager: success
F20260620 21:59:00.760145 1751 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63ca8ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f63ca8ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f63ca8ab4de google::LogMessage::Flush() at ??:0
@ 0x7f63ca8aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63cd56f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f63d5210989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f63d51ed023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f63d51ed54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f63d51e6652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f63d51e59e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f63d522c541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f63ccaee512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63ce1326db start_thread at ??:0
@ 0x7f63c93a961f clone at ??:0
W20260620 21:59:00.851617 1733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:00.851903 1733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:00.851977 1733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:00.856177 1733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:00.856258 1733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:00.856343 1733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:00.861694 1733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:42239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=36699
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:00.862797 1733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:00.863832 1733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:00.874356 1772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:00.874524 1733 server_base.cc:1034] running on GCE node
W20260620 21:59:00.874691 1771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:00.874342 1774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:00.875164 1733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:00.875926 1733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:00.877127 1733 hybrid_clock.cc:648] HybridClock initialized: now 1781992740877110 us; error 38 us; skew 500 ppm
Jun 20 21:59:00 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992740, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:00.880244 1733 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:59:00.881171 1733 webserver.cc:460] Webserver started at http://127.17.67.193:36699/ using document root <none> and password file <none>
I20260620 21:59:00.881812 1733 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:00.881904 1733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:00.886018 1733 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:00.888303 1781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:00.889685 1733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:00.889809 1733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "b907c73706b14b8180da1da77aaa181b"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "64ac235e269334a1f4be282e9a3475de"
server_key_iv: "460272550ff3db30110d0066a68796c1"
server_key_version: "encryptionkey@0"
I20260620 21:59:00.890385 1733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:00.915937 1733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:00.919333 1733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:00.919545 1733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:00.920156 1733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:00.921288 1788 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:59:00.926964 1733 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:59:00.927040 1733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 21:59:00.927140 1733 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:59:00.928735 1733 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:59:00.928807 1733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:59:00.929124 1788 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Bootstrap starting.
I20260620 21:59:00.939512 1733 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:42239
I20260620 21:59:00.939713 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:42239 every 8 connection(s)
I20260620 21:59:00.940541 1733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:00.944234 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1733
I20260620 21:59:00.944676 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:39975
--local_ip_for_outbound_sockets=127.17.67.194
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=40069
--webserver_interface=127.17.67.194
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:00.954021 1788 log.cc:824] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:00.961021 1788 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:59:00.961406 1788 tablet_bootstrap.cc:492] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Bootstrap complete.
I20260620 21:59:00.962615 1788 ts_tablet_manager.cc:1397] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.007s
I20260620 21:59:00.964922 1788 raft_consensus.cc:348] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:00.965359 1788 raft_consensus.cc:729] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b907c73706b14b8180da1da77aaa181b, State: Initialized, Role: FOLLOWER
I20260620 21:59:00.965849 1788 consensus_queue.cc:260] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:00.966773 1788 ts_tablet_manager.cc:1428] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 21:59:01.001070 1898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:00.942591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:42281)
0620 21:59:00.943078 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:00.943133 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:00.944034 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:01.000295 (+ 56261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
W20260620 21:59:01.001708 1896 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:44089 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:44089: Client connection negotiation failed: client connection to 127.17.67.254:44089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:59:01.002491 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.002244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:35355)
0620 21:59:01.002364 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.002436 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 21:59:01.003149 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.002947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:42855)
0620 21:59:01.003047 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.003093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:59:01.003351 1896 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:59:01.077035 1899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.077322 1899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.077401 1899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.081409 1899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.081498 1899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.081638 1899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:01.086897 1899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:39975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=40069
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:01.087956 1899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.088982 1899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:01.098416 1899 server_base.cc:1034] running on GCE node
W20260620 21:59:01.098477 1907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.098428 1909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.098403 1906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.099056 1899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.099663 1899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.100826 1899 hybrid_clock.cc:648] HybridClock initialized: now 1781992741100802 us; error 37 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.103564 1899 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:59:01.104437 1899 webserver.cc:460] Webserver started at http://127.17.67.194:40069/ using document root <none> and password file <none>
I20260620 21:59:01.104977 1899 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.105038 1899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.108775 1899 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:01.110834 1916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.112154 1899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:01.112327 1899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "2642ef2c427d471c9bd20ea7a8b6026e"
format_stamp: "Formatted at 2026-06-20 21:58:18 on dist-test-slave-cx8d"
server_key: "fc6c0b465067c934c739780f106cdf4a"
server_key_iv: "cfdd82c91d1ee704effbe3e00741af25"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.112916 1899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.138059 1899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.141858 1899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.142139 1899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.142815 1899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:01.143751 1899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:01.143832 1899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.143909 1899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:01.143975 1899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.153816 1899 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:39975
I20260620 21:59:01.153849 2029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:39975 every 8 connection(s)
I20260620 21:59:01.154938 1899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:01.159045 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.157137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:45933)
0620 21:59:01.157651 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.157770 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
W20260620 21:59:01.159539 2030 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:44089 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:44089: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
I20260620 21:59:01.160146 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.159917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:49303)
0620 21:59:01.160040 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.160091 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 21:59:01.160781 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.160582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:38615)
0620 21:59:01.160687 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.160730 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:59:01.161021 2030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:59:01.164742 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 1899
I20260620 21:59:01.165163 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:44049
--local_ip_for_outbound_sockets=127.17.67.195
--tserver_master_addrs=127.17.67.254:44089
--webserver_port=45849
--webserver_interface=127.17.67.195
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:01.277716 2033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:01.278014 2033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:01.278082 2033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:01.282119 2033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:01.282195 2033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:01.282277 2033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:01.287642 2033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:44049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=45849
--tserver_master_addrs=127.17.67.254:44089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:01.288740 2033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:01.289819 2033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:01.300262 2041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:01.300274 2039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.300336 2033 server_base.cc:1034] running on GCE node
W20260620 21:59:01.300274 2038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:01.301026 2033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:01.301762 2033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:01.302945 2033 hybrid_clock.cc:648] HybridClock initialized: now 1781992741302919 us; error 38 us; skew 500 ppm
Jun 20 21:59:01 dist-test-slave-cx8d krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992741, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:01.305749 2033 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:59:01.306576 2033 webserver.cc:460] Webserver started at http://127.17.67.195:45849/ using document root <none> and password file <none>
I20260620 21:59:01.307102 2033 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:01.307160 2033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:01.311043 2033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:01.313149 2048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.314502 2033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:01.314623 2033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "482336e07ccc42dd9ad4def8289c5a9f"
format_stamp: "Formatted at 2026-06-20 21:58:19 on dist-test-slave-cx8d"
server_key: "59c95267f9b04237152326e9617f75ab"
server_key_iv: "4668da90829a5cfa8c01ff3737cbbebc"
server_key_version: "encryptionkey@0"
I20260620 21:59:01.315164 2033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:01.328519 2033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:01.331641 2033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:01.331840 2033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:01.332414 2033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:01.333217 2033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:01.333262 2033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.333338 2033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:01.333377 2033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:01.342367 2033 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:44049
I20260620 21:59:01.342396 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:44049 every 8 connection(s)
I20260620 21:59:01.343397 2033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:01.347259 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.345496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:47911)
0620 21:59:01.345974 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.346082 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
W20260620 21:59:01.347759 2162 heartbeater.cc:644] Failed to heartbeat to 127.17.67.254:44089 (0 consecutive failures): Network error: Failed to ping master at 127.17.67.254:44089: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
I20260620 21:59:01.348450 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.348180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:37673)
0620 21:59:01.348313 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.348383 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 21:59:01.349148 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:01.348959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:36231)
0620 21:59:01.349057 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:01.349099 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:59:01.349391 2162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:59:01.351995 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2033
I20260620 21:59:02.004393 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.003938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:56655)
0620 21:59:02.004218 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:02.004280 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260620 21:59:02.162067 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.161665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:49593)
0620 21:59:02.161898 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:02.161957 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260620 21:59:02.350385 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.349973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:45565)
0620 21:59:02.350226 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:02.350287 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260620 21:59:02.528858 2169 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:02.529091 2169 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:02.530280 2169 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
Jun 20 21:59:02 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992740, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.194@KRBTEST.COM
Jun 20 21:59:02 dist-test-slave-cx8d krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992740, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.195@KRBTEST.COM
I20260620 21:59:02.546276 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.530530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44049 (local address 127.17.67.193:60191)
0620 21:59:02.531091 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:02.531112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:02.531240 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:02.532865 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:02.532871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:02.533498 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:02.534035 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:02.534056 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.535654 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.535664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:02.536778 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:02.536788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.537530 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.537534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:02.537771 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:02.538529 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:02.538544 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:02.540680 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:02.543049 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:02.543056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:02.543074 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:02.543615 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:02.544049 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:02.544053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:02.544055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:02.544352 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:02.545168 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:02.545175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:02.545486 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 21:59:02.545743 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 21:59:02.546088 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":172,"spinlock_wait_cycles":14720,"thread_start_us":90,"threads_started":1}
I20260620 21:59:02.546546 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.531125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:48055 (local address 127.17.67.194:39975)
0620 21:59:02.531434 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:02.531460 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:02.531491 (+ 31us) server_negotiation.cc:407] Connection header received
0620 21:59:02.532178 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:02.532184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:02.532489 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:02.532815 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:02.534223 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.535519 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.536916 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.537398 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.537615 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:02.540559 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:02.540564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:02.540576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:02.540606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:02.542896 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:02.543759 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:02.543766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:02.543772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:02.543944 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:02.544461 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:02.544465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:02.544467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:02.545023 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:02.545166 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:02.546017 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 21:59:02.546356 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":133,"threads_started":1}
I20260620 21:59:02.547051 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.530704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:60191 (local address 127.17.67.195:44049)
0620 21:59:02.531022 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:02.531055 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:02.531219 (+ 164us) server_negotiation.cc:407] Connection header received
0620 21:59:02.532085 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:02.532091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:02.532448 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:02.532814 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:02.534212 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.535499 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.536906 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.537401 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.537641 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:02.540872 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:02.540879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:02.540895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:02.540933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:02.542910 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:02.543759 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:02.543766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:02.543772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:02.543944 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:02.544461 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:02.544465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:02.544467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:02.545023 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:02.545166 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:02.546648 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0620 21:59:02.546924 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260620 21:59:02.547497 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:02.530903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39975 (local address 127.17.67.193:48055)
0620 21:59:02.531238 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:02.531252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:02.531343 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:02.532866 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:02.532872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:02.533499 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:02.534036 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:02.534051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.535671 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.535677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:02.536777 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:02.536787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:02.537527 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:02.537531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:02.537777 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:02.538479 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:02.538498 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:02.540315 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:02.543053 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:02.543056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:02.543074 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:02.543606 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:02.544069 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:02.544072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:02.544074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:02.544352 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:02.545414 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:02.545417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:02.545490 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 21:59:02.545743 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 21:59:02.547343 (+ 1600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1518,"spinlock_wait_cycles":39040,"thread_start_us":58,"threads_started":1}
W20260620 21:59:02.549543 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
W20260620 21:59:02.549835 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
I20260620 21:59:02.549907 1785 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b907c73706b14b8180da1da77aaa181b; no voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:59:02.550208 2169 raft_consensus.cc:2726] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:59:03.005453 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.005180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:52707)
0620 21:59:03.005309 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.005366 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:59:03.163167 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.162776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:43715)
0620 21:59:03.163013 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.163073 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260620 21:59:03.351681 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:03.351198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:41177)
0620 21:59:03.351477 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:03.351582 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260620 21:59:04.006927 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.006414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:51839)
0620 21:59:04.006757 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.006825 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260620 21:59:04.164599 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.164042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:53837)
0620 21:59:04.164379 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.164448 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":119,"threads_started":1}
I20260620 21:59:04.353012 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:04.352590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:59789)
0620 21:59:04.352862 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:04.352922 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260620 21:59:04.527146 2179 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:04.527285 2179 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:04.527809 2179 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
W20260620 21:59:04.528666 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
W20260620 21:59:04.528787 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
I20260620 21:59:04.528820 1785 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b907c73706b14b8180da1da77aaa181b; no voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:59:04.528954 2179 raft_consensus.cc:2726] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:59:05.008452 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.007955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:44991)
0620 21:59:05.008242 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.008300 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260620 21:59:05.165884 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.165454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:44257)
0620 21:59:05.165737 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.165791 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260620 21:59:05.354547 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:05.354071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:40239)
0620 21:59:05.354371 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:05.354447 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260620 21:59:06.010006 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:06.009428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:39727)
0620 21:59:06.009820 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:06.009886 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":152,"threads_started":1}
I20260620 21:59:06.167189 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:06.166745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:38779)
0620 21:59:06.167016 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:06.167074 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260620 21:59:06.355638 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:06.355252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:39807)
0620 21:59:06.355495 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:06.355551 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260620 21:59:07.011268 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:07.010848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:51693)
0620 21:59:07.011102 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:07.011165 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260620 21:59:07.168357 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:07.167921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:45657)
0620 21:59:07.168200 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:07.168260 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260620 21:59:07.356817 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:07.356372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:48689)
0620 21:59:07.356671 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:07.356727 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260620 21:59:07.802275 2189 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:07.802546 2189 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:07.802990 2189 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
W20260620 21:59:07.803854 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
W20260620 21:59:07.804024 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
I20260620 21:59:07.804114 1785 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b907c73706b14b8180da1da77aaa181b; no voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:59:07.804288 2189 raft_consensus.cc:2726] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:59:08.012662 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.012184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:32881)
0620 21:59:08.012455 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.012526 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260620 21:59:08.169754 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.169146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:48815)
0620 21:59:08.169432 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.169489 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":72,"threads_started":1}
I20260620 21:59:08.358076 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:08.357634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:42135)
0620 21:59:08.357927 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:08.357986 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":67,"threads_started":1}
I20260620 21:59:09.014053 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:09.013543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:41173)
0620 21:59:09.013885 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:09.013954 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":73,"threads_started":1}
I20260620 21:59:09.171078 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:09.170675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:35623)
0620 21:59:09.170920 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:09.170987 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260620 21:59:09.359225 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:09.358792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:60429)
0620 21:59:09.359071 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:09.359131 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260620 21:59:10.015309 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.014880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:47391)
0620 21:59:10.015141 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.015207 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260620 21:59:10.172194 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.171802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:44097)
0620 21:59:10.172044 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.172096 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260620 21:59:10.360419 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:10.359987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:44935)
0620 21:59:10.360259 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:10.360321 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260620 21:59:11.016954 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.016433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:45825)
0620 21:59:11.016786 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.016852 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20260620 21:59:11.173431 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.172981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:34651)
0620 21:59:11.173270 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.173328 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260620 21:59:11.340796 2201 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:11.340931 2201 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:11.341344 2201 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
W20260620 21:59:11.342415 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
W20260620 21:59:11.342585 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
I20260620 21:59:11.342664 1785 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b907c73706b14b8180da1da77aaa181b; no voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:59:11.342860 2201 raft_consensus.cc:2726] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:59:11.361771 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:11.361288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:56443)
0620 21:59:11.361558 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:11.361673 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260620 21:59:12.018249 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.017796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:37177)
0620 21:59:12.018087 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.018150 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260620 21:59:12.174646 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.174188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:44097)
0620 21:59:12.174462 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.174532 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260620 21:59:12.362998 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:12.362564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:40793)
0620 21:59:12.362827 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:12.362887 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260620 21:59:13.019721 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.019209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:56589)
0620 21:59:13.019503 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.019609 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260620 21:59:13.175871 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.175457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:49041)
0620 21:59:13.175719 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.175774 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260620 21:59:13.364310 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:13.363853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:47047)
0620 21:59:13.364158 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:13.364217 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260620 21:59:14.021035 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.020591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:39043)
0620 21:59:14.020869 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.020932 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260620 21:59:14.177004 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.176611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:51021)
0620 21:59:14.176846 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.176900 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260620 21:59:14.365496 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:14.365051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:38095)
0620 21:59:14.365333 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:14.365395 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260620 21:59:14.972033 2212 raft_consensus.cc:482] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:14.972239 2212 raft_consensus.cc:504] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482336e07ccc42dd9ad4def8289c5a9f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44049 } } peers { permanent_uuid: "b907c73706b14b8180da1da77aaa181b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 42239 } } peers { permanent_uuid: "2642ef2c427d471c9bd20ea7a8b6026e" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39975 } }
I20260620 21:59:14.972741 2212 leader_election.cc:288] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049), 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975)
W20260620 21:59:14.973887 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 482336e07ccc42dd9ad4def8289c5a9f (127.17.67.195:44049): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
W20260620 21:59:14.974049 1785 leader_election.cc:341] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2642ef2c427d471c9bd20ea7a8b6026e (127.17.67.194:39975): Not found: Tablet not found: f1ea04a9212d48358c948c32efd0a75c
I20260620 21:59:14.974120 1785 leader_election.cc:302] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b907c73706b14b8180da1da77aaa181b; no voters: 2642ef2c427d471c9bd20ea7a8b6026e, 482336e07ccc42dd9ad4def8289c5a9f
I20260620 21:59:14.974292 2212 raft_consensus.cc:2726] T f1ea04a9212d48358c948c32efd0a75c P b907c73706b14b8180da1da77aaa181b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:59:15.022410 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.021978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:49113)
0620 21:59:15.022250 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.022304 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260620 21:59:15.178205 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.177796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:35577)
0620 21:59:15.178056 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.178111 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260620 21:59:15.366812 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:15.366340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:38413)
0620 21:59:15.366648 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:15.366713 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20260620 21:59:16.023901 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.023443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.193:45061)
0620 21:59:16.023735 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.023798 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260620 21:59:16.179488 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.179058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.194:35113)
0620 21:59:16.179328 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.179388 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260620 21:59:16.361157 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1733
I20260620 21:59:16.367239 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 1899
I20260620 21:59:16.368778 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:16.368247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:44089 (local address 127.17.67.195:47407)
0620 21:59:16.368565 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:16.368639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.254:44089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
I20260620 21:59:16.372896 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2033
2026-06-20T21:59:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59348 ms)
[----------] 30 tests from SecurityITest (243123 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:16 dist-test-slave-cx8d krb5kdc[2221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:16 dist-test-slave-cx8d krb5kdc[2221](info): set up 2 sockets
Jun 20 21:59:16 dist-test-slave-cx8d krb5kdc[2221](info): commencing operation
krb5kdc: starting...
W20260620 21:59:17.481428 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:17 dist-test-slave-cx8d krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:17.604301 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:37095
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:37095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:17.716607 2234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:17.716873 2234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:17.716912 2234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:17.721040 2234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:17.721097 2234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:17.721118 2234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:17.721135 2234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:17.721197 2234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:17.726611 2234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:37095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:17.727725 2234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:17.728773 2234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:17.737372 2240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:17.737375 2239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:17.737440 2234 server_base.cc:1034] running on GCE node
W20260620 21:59:17.737372 2242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:17.738236 2234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:17.739383 2234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:17.740585 2234 hybrid_clock.cc:648] HybridClock initialized: now 1781992757740543 us; error 51 us; skew 500 ppm
Jun 20 21:59:17 dist-test-slave-cx8d krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:17.743389 2234 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:17.744318 2234 webserver.cc:460] Webserver started at http://127.17.67.254:36425/ using document root <none> and password file <none>
I20260620 21:59:17.744886 2234 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:17.744978 2234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:17.745215 2234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:17.747330 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "4a042b05d27749f386a2be8c0e10dcfc"
format_stamp: "Formatted at 2026-06-20 21:59:17 on dist-test-slave-cx8d"
server_key: "aca033e02426c0d3263055212e8656c1"
server_key_iv: "8fc3104db149120fe8d1160e0b27e221"
server_key_version: "encryptionkey@0"
I20260620 21:59:17.747918 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "4a042b05d27749f386a2be8c0e10dcfc"
format_stamp: "Formatted at 2026-06-20 21:59:17 on dist-test-slave-cx8d"
server_key: "aca033e02426c0d3263055212e8656c1"
server_key_iv: "8fc3104db149120fe8d1160e0b27e221"
server_key_version: "encryptionkey@0"
I20260620 21:59:17.751715 2234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 21:59:17.754428 2249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:17.755846 2234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:17.755988 2234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "4a042b05d27749f386a2be8c0e10dcfc"
format_stamp: "Formatted at 2026-06-20 21:59:17 on dist-test-slave-cx8d"
server_key: "aca033e02426c0d3263055212e8656c1"
server_key_iv: "8fc3104db149120fe8d1160e0b27e221"
server_key_version: "encryptionkey@0"
I20260620 21:59:17.756117 2234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:17.778771 2234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:17.782814 2234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:17.783022 2234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:17.791247 2234 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:37095
I20260620 21:59:17.791324 2301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:37095 every 8 connection(s)
I20260620 21:59:17.792241 2234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:17.796029 2302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:17.802012 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2234
I20260620 21:59:17.802157 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:17.802451 17679 external_mini_cluster.cc:1428] Setting key 868a19ca0e0ceaf90c1a7f0b04ac7ceb
I20260620 21:59:17.804566 2302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: Bootstrap starting.
I20260620 21:59:17.807752 2302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:17.808784 2302 log.cc:824] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:59:17 dist-test-slave-cx8d krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:17.811283 2302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: No bootstrap required, opened a new log
I20260620 21:59:17.814620 2302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } }
I20260620 21:59:17.814829 2302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:17.814905 2302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a042b05d27749f386a2be8c0e10dcfc, State: Initialized, Role: FOLLOWER
I20260620 21:59:17.815371 2302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } }
I20260620 21:59:17.815524 2302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:17.815587 2302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:17.815680 2302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:17.816767 2302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } }
I20260620 21:59:17.817124 2302 leader_election.cc:302] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a042b05d27749f386a2be8c0e10dcfc; no voters:
I20260620 21:59:17.817431 2302 leader_election.cc:288] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:17.817535 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:17.803849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.17.67.254:37095)
0620 21:59:17.804272 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:17.804292 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:17.804339 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:59:17.805028 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:17.805046 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:17.805377 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:17.805753 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:17.806552 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.807563 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.808528 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:17.809059 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:17.809220 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:17.811921 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:17.811926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:17.811941 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:17.811968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:17.814223 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.814680 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.814685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.814690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.814796 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:17.815150 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:17.815155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:17.815157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:17.815594 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:17.815797 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:17.816218 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 21:59:17.816529 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260620 21:59:17.817559 2307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:17.817829 2307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [term 1 LEADER]: Becoming Leader. State: Replica: 4a042b05d27749f386a2be8c0e10dcfc, State: Running, Role: LEADER
I20260620 21:59:17.818140 2307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } }
I20260620 21:59:17.818607 2302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:17.819257 2307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a042b05d27749f386a2be8c0e10dcfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } } }
I20260620 21:59:17.819362 2307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:17.819546 2307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a042b05d27749f386a2be8c0e10dcfc. Latest consensus state: current_term: 1 leader_uuid: "4a042b05d27749f386a2be8c0e10dcfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a042b05d27749f386a2be8c0e10dcfc" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 37095 } } }
I20260620 21:59:17.819625 2307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:17.820461 2315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:17.823748 2315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:17.829677 2315 catalog_manager.cc:1347] Generated new cluster ID: be21e399b8784144a8f10b424d249b92
I20260620 21:59:17.829751 2315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:17.839308 2315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:17.840525 2315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:17.846367 2315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: Generated new TSK 0
I20260620 21:59:17.847074 2315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:59:17.878544 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:18.001662 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:18.001905 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:18.001947 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:18.005892 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:18.005941 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:18.005997 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:18.010993 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:18.011935 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:18.012869 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:18.022758 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:18.022770 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:18.022918 2328 server_base.cc:1034] running on GCE node
W20260620 21:59:18.023149 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:18.023571 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:18.024217 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:18.025414 2328 hybrid_clock.cc:648] HybridClock initialized: now 1781992758025378 us; error 47 us; skew 500 ppm
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:18.028267 2328 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:59:18.029172 2328 webserver.cc:460] Webserver started at http://127.17.67.193:34657/ using document root <none> and password file <none>
I20260620 21:59:18.029810 2328 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:18.029901 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:18.030130 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:18.032109 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "11351e993a2f449dbe5f8997a0432f67"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "0ab87aa90a768187e861c4b743a0880e"
server_key_iv: "1fec91ba04463517d6010f7443d9e9ed"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.032660 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "11351e993a2f449dbe5f8997a0432f67"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "0ab87aa90a768187e861c4b743a0880e"
server_key_iv: "1fec91ba04463517d6010f7443d9e9ed"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.036401 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 21:59:18.039135 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.040334 2328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:18.040511 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "11351e993a2f449dbe5f8997a0432f67"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "0ab87aa90a768187e861c4b743a0880e"
server_key_iv: "1fec91ba04463517d6010f7443d9e9ed"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.040628 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:18.076869 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:18.080133 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:18.080353 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:18.080967 2328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:18.081831 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:18.081907 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.081974 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:18.082028 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.091096 2328 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:44027
I20260620 21:59:18.091120 2456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:44027 every 8 connection(s)
I20260620 21:59:18.092023 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:18.095629 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2328
I20260620 21:59:18.095762 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:18.096060 17679 external_mini_cluster.cc:1428] Setting key 20925083205cabadc24bee9d698aa224
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:59:18.106797 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.093748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:46273 (local address 127.17.67.254:37095)
0620 21:59:18.093884 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.093888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.094801 (+ 913us) server_negotiation.cc:407] Connection header received
0620 21:59:18.096356 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.096360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.096418 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.096523 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:18.097739 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.098286 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.099982 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.100219 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.100304 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.103199 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:18.103209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:18.103212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:18.103236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:18.104697 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.105220 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.105223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.105225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.105321 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.105691 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.105694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.105696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.105878 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:18.105980 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.106482 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.106619 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:59:18.107625 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.094184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:37095 (local address 127.17.67.193:46273)
0620 21:59:18.094635 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.094679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.096124 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.096703 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.096710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.097056 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:18.097538 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.097551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.098425 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.098429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.099846 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.099861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.100381 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.100387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.100627 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.101286 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:18.101311 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:18.103031 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:18.104832 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.104838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.104853 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.105113 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.105411 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.105414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.105416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.105603 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.105990 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:18.105999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:18.106248 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.106428 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.106756 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:59:18.109046 2457 heartbeater.cc:347] Connected to a master server at 127.17.67.254:37095
I20260620 21:59:18.109300 2457 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:18.109884 2457 heartbeater.cc:511] Master 127.17.67.254:37095 requested a full tablet report, sending...
I20260620 21:59:18.111359 2266 ts_manager.cc:194] Registered new tserver with Master: 11351e993a2f449dbe5f8997a0432f67 (127.17.67.193:44027)
I20260620 21:59:18.112468 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:46273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:59:18.125648 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:18.235673 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:18.235917 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:18.235960 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:18.239851 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:18.239904 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:18.239960 2462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:18.245157 2462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:18.246173 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:18.247113 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:18.256646 2467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:18.256702 2468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:18.256630 2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:18.256896 2462 server_base.cc:1034] running on GCE node
I20260620 21:59:18.257279 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:18.257964 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:18.259145 2462 hybrid_clock.cc:648] HybridClock initialized: now 1781992758259130 us; error 30 us; skew 500 ppm
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:18.262161 2462 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:59:18.262929 2462 webserver.cc:460] Webserver started at http://127.17.67.194:32773/ using document root <none> and password file <none>
I20260620 21:59:18.263437 2462 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:18.263494 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:18.263651 2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:18.265538 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "f31a00a815b5404dac4d32f65c3115d6"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "aecba1ebae9322e82a9959b66c8a81cc"
server_key_iv: "11170e530d755771fba59a4559a8f2b9"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.266093 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "f31a00a815b5404dac4d32f65c3115d6"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "aecba1ebae9322e82a9959b66c8a81cc"
server_key_iv: "11170e530d755771fba59a4559a8f2b9"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.269517 2462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:18.271903 2477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.273103 2462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:18.273195 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "f31a00a815b5404dac4d32f65c3115d6"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "aecba1ebae9322e82a9959b66c8a81cc"
server_key_iv: "11170e530d755771fba59a4559a8f2b9"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.273425 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:18.288028 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:18.291116 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:18.291309 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:18.291913 2462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:18.292692 2462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:18.292744 2462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.292810 2462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:18.292848 2462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.301707 2462 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:41339
I20260620 21:59:18.301793 2590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:41339 every 8 connection(s)
I20260620 21:59:18.302572 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:18.312250 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2462
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:18.312484 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:18.312798 17679 external_mini_cluster.cc:1428] Setting key 84e18bc184b908c200b3739c46a0abe6
I20260620 21:59:18.317195 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.304354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:42625 (local address 127.17.67.254:37095)
0620 21:59:18.304503 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.304506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.305292 (+ 786us) server_negotiation.cc:407] Connection header received
0620 21:59:18.306785 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.306789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.306875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.306993 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:18.308196 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.308791 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.310214 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.310460 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.310586 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.313066 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:18.313070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:18.313073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:18.313100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:18.314882 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.315348 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.315355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.315360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.315459 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.315780 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.315786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.315791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.316046 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:18.316157 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.316826 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.316993 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:59:18.318185 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.304683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:37095 (local address 127.17.67.194:42625)
0620 21:59:18.305120 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.305165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.306561 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.307177 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.307185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.307551 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:18.308035 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.308048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.308937 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.308942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.310046 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.310056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.310619 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.310623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.310800 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.311316 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:18.311333 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:18.312897 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:18.314990 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.314995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.315006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.315250 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.315567 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.315569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.315570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.315691 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.316165 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:18.316175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:18.316497 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.316750 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.317086 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260620 21:59:18.319480 2591 heartbeater.cc:347] Connected to a master server at 127.17.67.254:37095
I20260620 21:59:18.319717 2591 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:18.320286 2591 heartbeater.cc:511] Master 127.17.67.254:37095 requested a full tablet report, sending...
I20260620 21:59:18.321468 2266 ts_manager.cc:194] Registered new tserver with Master: f31a00a815b5404dac4d32f65c3115d6 (127.17.67.194:41339)
I20260620 21:59:18.322309 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:42625
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:59:18.340926 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:18.448802 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:18.449101 2596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:18.449177 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:18.453372 2596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:18.453449 2596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:18.453537 2596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:18.458819 2596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:18.459851 2596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:18.460889 2596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:18.470230 2602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:18.470249 2604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:18.470376 2596 server_base.cc:1034] running on GCE node
W20260620 21:59:18.470230 2601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:18.470929 2596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:18.471587 2596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:18.472813 2596 hybrid_clock.cc:648] HybridClock initialized: now 1781992758472803 us; error 29 us; skew 500 ppm
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:18.475404 2596 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:59:18.476291 2596 webserver.cc:460] Webserver started at http://127.17.67.195:42615/ using document root <none> and password file <none>
I20260620 21:59:18.476858 2596 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:18.476918 2596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:18.477133 2596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:18.479148 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "302b508e4a64441f44028033bf8cbade"
server_key_iv: "aa7ec1ee6f24fba4fa3f6a80c91a41a3"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.479727 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "302b508e4a64441f44028033bf8cbade"
server_key_iv: "aa7ec1ee6f24fba4fa3f6a80c91a41a3"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.483332 2596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:18.485621 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.486884 2596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:18.486976 2596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d"
format_stamp: "Formatted at 2026-06-20 21:59:18 on dist-test-slave-cx8d"
server_key: "302b508e4a64441f44028033bf8cbade"
server_key_iv: "aa7ec1ee6f24fba4fa3f6a80c91a41a3"
server_key_version: "encryptionkey@0"
I20260620 21:59:18.487058 2596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:18.505174 2596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:18.508235 2596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:18.508402 2596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:18.508920 2596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:18.509691 2596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:18.509737 2596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.509774 2596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:18.509789 2596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:18.518891 2596 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:44123
I20260620 21:59:18.518913 2724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:44123 every 8 connection(s)
I20260620 21:59:18.519970 2596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:18.527351 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2596
I20260620 21:59:18.527527 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:18.527833 17679 external_mini_cluster.cc:1428] Setting key 1a017aa4604e6e356e28aa1995a690f4
Jun 20 21:59:18 dist-test-slave-cx8d krb5kdc[2221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992758, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:18.534292 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.521876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:54109 (local address 127.17.67.254:37095)
0620 21:59:18.522014 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.522019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.522917 (+ 898us) server_negotiation.cc:407] Connection header received
0620 21:59:18.524285 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.524288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.524344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.524457 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:18.525652 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.526189 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.527578 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.527820 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.527963 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.530924 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:18.530926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:18.530928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:18.530952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:18.532263 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.532781 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.532784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.532785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.532855 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.533245 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.533247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.533249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.533451 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:18.533538 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.534076 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.534175 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:59:18.535168 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.522307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:37095 (local address 127.17.67.195:54109)
0620 21:59:18.522766 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.522809 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.524069 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.524645 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.524653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.525001 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:18.525480 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.525494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.526315 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.526319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.527347 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.527359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.528019 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.528024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.528217 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.528918 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:18.528938 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:18.530772 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:18.532381 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.532386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.532397 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.532694 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.532976 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:18.532978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:18.532980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:18.533120 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:18.533556 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:18.533561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:18.533854 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.534018 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.534279 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260620 21:59:18.536285 2725 heartbeater.cc:347] Connected to a master server at 127.17.67.254:37095
I20260620 21:59:18.536653 2725 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:18.537142 2725 heartbeater.cc:511] Master 127.17.67.254:37095 requested a full tablet report, sending...
I20260620 21:59:18.538223 2266 ts_manager.cc:194] Registered new tserver with Master: 2ecaea4940f846ba8b6775e8dbf2cd4d (127.17.67.195:44123)
I20260620 21:59:18.538861 2266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:54109
I20260620 21:59:18.541797 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:18.551790 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.543756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.17.67.254:37095)
0620 21:59:18.543887 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.543892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.544074 (+ 182us) server_negotiation.cc:407] Connection header received
0620 21:59:18.544183 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.544185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.544221 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.544284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:18.545144 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.545758 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.546616 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.546818 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.546899 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.548487 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:18.548490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:18.548492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:18.548516 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:18.550024 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.550427 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.550430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.550431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.550509 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:18.550824 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:18.550827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:18.550828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:18.551072 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:18.551187 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.551457 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.551593 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:59:18.554970 2266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:18.558353 2266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:18.573480 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.567206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:44027 (local address 127.0.0.1:49740)
0620 21:59:18.567572 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.567625 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.567745 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.568613 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.568623 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.568652 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:18.568813 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.568821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.570234 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.570241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.571619 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.571631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.572884 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.572888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.573005 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.573051 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.573166 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.573287 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":8,"thread_start_us":84,"threads_started":1}
I20260620 21:59:18.573544 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.567358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.17.67.193:44027)
0620 21:59:18.567659 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.567679 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.567706 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:59:18.567943 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.567949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.568200 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.568396 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:18.568982 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.570095 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.571772 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.572755 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.572874 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.573034 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.573220 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.573353 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260620 21:59:18.574540 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.567985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44123 (local address 127.0.0.1:52190)
0620 21:59:18.568268 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.568284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.568394 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.569152 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.569156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.569173 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:18.569280 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.569289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.570894 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.570898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.572184 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.572195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.573400 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.573404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.574147 (+ 743us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.574214 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.574284 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.574413 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":142,"thread_start_us":94,"threads_started":1}
I20260620 21:59:18.574687 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.568944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.17.67.194:41339)
0620 21:59:18.569421 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.569434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.569453 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:59:18.569522 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.569527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.569727 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.569877 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:18.570109 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.571102 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.572534 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.573287 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.573521 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.573736 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.574405 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.574537 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":118,"threads_started":1}
I20260620 21:59:18.574580 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.568171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.17.67.195:44123)
0620 21:59:18.568703 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.568716 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.568738 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:18.568815 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.568820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.568994 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.569432 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:18.569599 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.570735 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.572331 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.573253 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.574119 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.574262 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.574319 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.574420 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":172,"threads_started":1}
I20260620 21:59:18.575170 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.568324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:41339 (local address 127.0.0.1:35352)
0620 21:59:18.568647 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.568660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.568744 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.569876 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.569880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.569900 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:18.569991 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.569996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.571224 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.571228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.572411 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.572419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.573414 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.573417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.574171 (+ 754us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.574213 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.574287 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.575041 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1262,"thread_start_us":161,"threads_started":1}
I20260620 21:59:18.576941 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 69393ddbd52d45129091424941bce1a6 (DEFAULT_TABLE table=test-table [id=0c2d181a37dd4a5bad84be4425035044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:18.578044 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69393ddbd52d45129091424941bce1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:18.577745 2659 tablet_service.cc:1467] Processing CreateTablet for tablet 69393ddbd52d45129091424941bce1a6 (DEFAULT_TABLE table=test-table [id=0c2d181a37dd4a5bad84be4425035044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:18.577745 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 69393ddbd52d45129091424941bce1a6 (DEFAULT_TABLE table=test-table [id=0c2d181a37dd4a5bad84be4425035044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:18.578734 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69393ddbd52d45129091424941bce1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:18.578734 2659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69393ddbd52d45129091424941bce1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:18.584879 2742 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Bootstrap starting.
I20260620 21:59:18.586519 2744 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Bootstrap starting.
I20260620 21:59:18.587457 2742 tablet_bootstrap.cc:654] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:18.588927 2744 tablet_bootstrap.cc:654] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:18.588919 2742 log.cc:824] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:18.589547 2743 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Bootstrap starting.
I20260620 21:59:18.590310 2744 log.cc:824] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:18.591197 2742 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: No bootstrap required, opened a new log
I20260620 21:59:18.591428 2742 ts_tablet_manager.cc:1397] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:59:18.591666 2743 tablet_bootstrap.cc:654] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:18.592342 2744 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: No bootstrap required, opened a new log
I20260620 21:59:18.592561 2744 ts_tablet_manager.cc:1397] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:18.592845 2743 log.cc:824] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:18.594796 2743 tablet_bootstrap.cc:492] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: No bootstrap required, opened a new log
I20260620 21:59:18.594967 2743 ts_tablet_manager.cc:1397] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:59:18.595412 2742 raft_consensus.cc:348] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.595672 2742 raft_consensus.cc:374] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:18.595902 2742 raft_consensus.cc:729] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11351e993a2f449dbe5f8997a0432f67, State: Initialized, Role: FOLLOWER
I20260620 21:59:18.596378 2742 consensus_queue.cc:260] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.596251 2744 raft_consensus.cc:348] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.596529 2744 raft_consensus.cc:374] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:18.596586 2744 raft_consensus.cc:729] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31a00a815b5404dac4d32f65c3115d6, State: Initialized, Role: FOLLOWER
I20260620 21:59:18.597075 2744 consensus_queue.cc:260] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.597429 2742 ts_tablet_manager.cc:1428] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:59:18.597807 2457 heartbeater.cc:503] Master 127.17.67.254:37095 was elected leader, sending a full tablet report...
I20260620 21:59:18.597807 2591 heartbeater.cc:503] Master 127.17.67.254:37095 was elected leader, sending a full tablet report...
I20260620 21:59:18.598436 2743 raft_consensus.cc:348] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.598647 2743 raft_consensus.cc:374] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:18.598745 2743 raft_consensus.cc:729] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecaea4940f846ba8b6775e8dbf2cd4d, State: Initialized, Role: FOLLOWER
I20260620 21:59:18.599126 2743 consensus_queue.cc:260] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.599594 2744 ts_tablet_manager.cc:1428] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:59:18.599988 2743 ts_tablet_manager.cc:1428] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 21:59:18.600335 2725 heartbeater.cc:503] Master 127.17.67.254:37095 was elected leader, sending a full tablet report...
W20260620 21:59:18.770917 2726 tablet.cc:2369] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:18.786799 2749 raft_consensus.cc:482] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:18.787029 2749 raft_consensus.cc:504] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.788151 2749 leader_election.cc:288] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11351e993a2f449dbe5f8997a0432f67 (127.17.67.193:44027), 2ecaea4940f846ba8b6775e8dbf2cd4d (127.17.67.195:44123)
I20260620 21:59:18.792750 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.788881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:32985 (local address 127.17.67.195:44123)
0620 21:59:18.789025 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.789030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.789150 (+ 120us) server_negotiation.cc:407] Connection header received
0620 21:59:18.789250 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.789252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.789319 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.789396 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:18.789687 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.790386 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.791776 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.792179 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.792308 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.792361 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.792549 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.792623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:59:18.792860 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.788430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:44027 (local address 127.17.67.194:34529)
0620 21:59:18.788603 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.788620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.788736 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.789052 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.789056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.789082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:18.789226 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.789235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.790266 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.790268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.791617 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.791628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.792453 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.792458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.792582 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.792600 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.792666 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.792730 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":51}
I20260620 21:59:18.792882 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.788507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:34529 (local address 127.17.67.193:44027)
0620 21:59:18.788642 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.788649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.788671 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:18.788888 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.788891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.788945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.789023 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:18.789343 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.790172 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.791770 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.792342 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.792469 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.792514 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.792695 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.792766 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:59:18.793098 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.788755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:44123 (local address 127.17.67.194:32985)
0620 21:59:18.789038 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.789062 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:18.789158 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:18.789426 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:18.789428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:18.789445 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:18.789543 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.789547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.790474 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.790477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.791641 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:18.791647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.792300 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.792303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:18.792403 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.792435 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:59:18.792876 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 21:59:18.792949 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260620 21:59:18.793449 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69393ddbd52d45129091424941bce1a6" candidate_uuid: "f31a00a815b5404dac4d32f65c3115d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11351e993a2f449dbe5f8997a0432f67" is_pre_election: true
I20260620 21:59:18.793747 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69393ddbd52d45129091424941bce1a6" candidate_uuid: "f31a00a815b5404dac4d32f65c3115d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" is_pre_election: true
I20260620 21:59:18.793773 2411 raft_consensus.cc:2454] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f31a00a815b5404dac4d32f65c3115d6 in term 0.
I20260620 21:59:18.794026 2679 raft_consensus.cc:2454] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f31a00a815b5404dac4d32f65c3115d6 in term 0.
I20260620 21:59:18.794282 2479 leader_election.cc:302] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11351e993a2f449dbe5f8997a0432f67, f31a00a815b5404dac4d32f65c3115d6; no voters:
I20260620 21:59:18.794580 2749 raft_consensus.cc:2781] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:18.794674 2749 raft_consensus.cc:482] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:18.794724 2749 raft_consensus.cc:3037] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:18.795817 2749 raft_consensus.cc:504] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.796149 2749 leader_election.cc:288] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [CANDIDATE]: Term 1 election: Requested vote from peers 11351e993a2f449dbe5f8997a0432f67 (127.17.67.193:44027), 2ecaea4940f846ba8b6775e8dbf2cd4d (127.17.67.195:44123)
I20260620 21:59:18.796518 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69393ddbd52d45129091424941bce1a6" candidate_uuid: "f31a00a815b5404dac4d32f65c3115d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d"
I20260620 21:59:18.796554 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69393ddbd52d45129091424941bce1a6" candidate_uuid: "f31a00a815b5404dac4d32f65c3115d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11351e993a2f449dbe5f8997a0432f67"
I20260620 21:59:18.796653 2679 raft_consensus.cc:3037] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:18.796665 2411 raft_consensus.cc:3037] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:18.797711 2679 raft_consensus.cc:2454] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f31a00a815b5404dac4d32f65c3115d6 in term 1.
I20260620 21:59:18.797938 2411 raft_consensus.cc:2454] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f31a00a815b5404dac4d32f65c3115d6 in term 1.
I20260620 21:59:18.797992 2479 leader_election.cc:302] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecaea4940f846ba8b6775e8dbf2cd4d, f31a00a815b5404dac4d32f65c3115d6; no voters:
I20260620 21:59:18.798153 2749 raft_consensus.cc:2781] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:18.798436 2749 raft_consensus.cc:686] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 1 LEADER]: Becoming Leader. State: Replica: f31a00a815b5404dac4d32f65c3115d6, State: Running, Role: LEADER
I20260620 21:59:18.798836 2749 consensus_queue.cc:237] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } }
I20260620 21:59:18.802342 2264 catalog_manager.cc:5495] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f31a00a815b5404dac4d32f65c3115d6 (127.17.67.194). New cstate: current_term: 1 leader_uuid: "f31a00a815b5404dac4d32f65c3115d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f31a00a815b5404dac4d32f65c3115d6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 41339 } health_report { overall_health: HEALTHY } } }
W20260620 21:59:18.803633 2592 tablet.cc:2369] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:18.843327 2458 tablet.cc:2369] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:18.864274 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.860633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.17.67.194:41339)
0620 21:59:18.860771 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:18.860775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:18.860789 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:18.860892 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:18.860894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:18.860951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:18.861040 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:18.861315 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.861966 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.862787 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:18.863020 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:18.863105 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:18.863365 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:18.863830 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:18.863936 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:18.864001 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:59:18.864051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:59:18.870208 2679 raft_consensus.cc:1261] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 1 FOLLOWER]: Refusing update from remote peer f31a00a815b5404dac4d32f65c3115d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:18.870285 2411 raft_consensus.cc:1261] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 1 FOLLOWER]: Refusing update from remote peer f31a00a815b5404dac4d32f65c3115d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:18.870908 2749 consensus_queue.cc:1035] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecaea4940f846ba8b6775e8dbf2cd4d" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:18.871132 2752 consensus_queue.cc:1035] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11351e993a2f449dbe5f8997a0432f67" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 44027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:18.878326 2755 mvcc.cc:205] Tried to move back new op lower bound from 7299042340325683200 to 7299042340043513856. Current Snapshot: MvccSnapshot[applied={T|T < 7299042340325683200}]
I20260620 21:59:18.881181 2757 mvcc.cc:205] Tried to move back new op lower bound from 7299042340325683200 to 7299042340043513856. Current Snapshot: MvccSnapshot[applied={T|T < 7299042340325683200 or (T in {7299042340325683200})}]
I20260620 21:59:18.888747 2266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:18.888931 2266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:18.891551 2266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a042b05d27749f386a2be8c0e10dcfc: Sending DeleteTablet for 3 replicas of tablet 69393ddbd52d45129091424941bce1a6
I20260620 21:59:18.892333 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 69393ddbd52d45129091424941bce1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:18 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:49740
I20260620 21:59:18.892532 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 69393ddbd52d45129091424941bce1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:18 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:35352
I20260620 21:59:18.892853 2765 tablet_replica.cc:330] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67: stopping tablet replica
I20260620 21:59:18.892877 2659 tablet_service.cc:1514] Processing DeleteTablet for tablet 69393ddbd52d45129091424941bce1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:18 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:52190
I20260620 21:59:18.893162 2765 raft_consensus.cc:2229] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:18.893388 2767 tablet_replica.cc:330] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: stopping tablet replica
I20260620 21:59:18.893484 2765 raft_consensus.cc:2258] T 69393ddbd52d45129091424941bce1a6 P 11351e993a2f449dbe5f8997a0432f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:18.893821 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2328
I20260620 21:59:18.893813 2767 raft_consensus.cc:2229] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:18.894086 2767 raft_consensus.cc:2258] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:18.895722 2767 ts_tablet_manager.cc:1905] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:18.899237 2767 ts_tablet_manager.cc:1918] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:18.899344 2767 log.cc:1196] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/69393ddbd52d45129091424941bce1a6
I20260620 21:59:18.899609 2766 tablet_replica.cc:330] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: stopping tablet replica
I20260620 21:59:18.899689 2767 ts_tablet_manager.cc:1939] T 69393ddbd52d45129091424941bce1a6 P 2ecaea4940f846ba8b6775e8dbf2cd4d: Deleting consensus metadata
I20260620 21:59:18.899995 2766 raft_consensus.cc:2229] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:18.900414 2766 raft_consensus.cc:2258] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:18.901002 2251 catalog_manager.cc:4841] TS 2ecaea4940f846ba8b6775e8dbf2cd4d (127.17.67.195:44123): tablet 69393ddbd52d45129091424941bce1a6 (table test-table [id=0c2d181a37dd4a5bad84be4425035044]) successfully deleted
I20260620 21:59:18.901954 2766 ts_tablet_manager.cc:1905] T 69393ddbd52d45129091424941bce1a6 P f31a00a815b5404dac4d32f65c3115d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:59:18.902890 2251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:44027 (error 108)
I20260620 21:59:18.903923 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.903610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:44027 (local address 127.0.0.1:49748)
0620 21:59:18.903743 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.903820 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:44027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:59:18.904050 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2462
W20260620 21:59:18.904217 2251 catalog_manager.cc:4584] TS 11351e993a2f449dbe5f8997a0432f67 (127.17.67.193:44027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69393ddbd52d45129091424941bce1a6: Network error: Client connection negotiation failed: client connection to 127.17.67.193:44027: connect: Connection refused (error 111)
I20260620 21:59:18.909837 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2596
I20260620 21:59:18.910722 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:18.909965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:41339 (local address 127.0.0.1:35370)
0620 21:59:18.910543 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:18.910611 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.194:41339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462}
I20260620 21:59:18.915196 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2234
2026-06-20T21:59:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2541 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T21:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:18Z Disabled control of system clock
I20260620 21:59:18.946190 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:33751
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:33751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:19.054610 2772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:19.054868 2772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:19.054904 2772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:19.059502 2772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:19.059571 2772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:19.059587 2772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:19.059603 2772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:19.059619 2772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:19.064409 2772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:33751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:33751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:19.065397 2772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:19.066772 2772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:19.075101 2780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.075111 2777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.075111 2778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.075418 2772 server_base.cc:1034] running on GCE node
I20260620 21:59:19.076071 2772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:19.077050 2772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:19.078205 2772 hybrid_clock.cc:648] HybridClock initialized: now 1781992759078186 us; error 32 us; skew 500 ppm
I20260620 21:59:19.079972 2772 webserver.cc:460] Webserver started at http://127.17.67.254:46567/ using document root <none> and password file <none>
I20260620 21:59:19.080513 2772 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:19.080575 2772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:19.080801 2772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:19.082804 2772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "2874af00571e4e1bb5fc4eeade10e9db"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "873f8ccc47fa889fe7a4c1706625a326"
server_key_iv: "4a46e858497c67f96b5d3d8d14a81343"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.083353 2772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "2874af00571e4e1bb5fc4eeade10e9db"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "873f8ccc47fa889fe7a4c1706625a326"
server_key_iv: "4a46e858497c67f96b5d3d8d14a81343"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.086819 2772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:19.089105 2786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.090335 2772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:19.090466 2772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "2874af00571e4e1bb5fc4eeade10e9db"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "873f8ccc47fa889fe7a4c1706625a326"
server_key_iv: "4a46e858497c67f96b5d3d8d14a81343"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.090572 2772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:19.111819 2772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:19.112510 2772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:19.112704 2772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:19.119285 2772 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:33751
I20260620 21:59:19.119292 2838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:33751 every 8 connection(s)
I20260620 21:59:19.120221 2772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:19.121742 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2772
I20260620 21:59:19.121866 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:19.122212 17679 external_mini_cluster.cc:1428] Setting key ad15a6e66dd0a2b5cd8eeb5a4c0f890c
I20260620 21:59:19.123879 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:19.129207 2839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: Bootstrap starting.
I20260620 21:59:19.131237 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.123578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.17.67.254:33751)
0620 21:59:19.124234 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.124250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.124285 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:19.124930 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.124946 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.125332 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.125688 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:19.126088 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.127022 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.127875 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.128353 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.128522 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.129518 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:19.129523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:19.129535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:19.129972 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:19.130027 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:19.130047 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:19.130155 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.130275 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.130523 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260620 21:59:19.131565 2839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:19.132334 2839 log.cc:824] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:19.134330 2839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: No bootstrap required, opened a new log
I20260620 21:59:19.138149 2839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } }
I20260620 21:59:19.138355 2839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:19.138394 2839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2874af00571e4e1bb5fc4eeade10e9db, State: Initialized, Role: FOLLOWER
I20260620 21:59:19.138787 2839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } }
I20260620 21:59:19.138908 2839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:19.138960 2839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:19.139029 2839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:19.139884 2839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } }
I20260620 21:59:19.140161 2839 leader_election.cc:302] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2874af00571e4e1bb5fc4eeade10e9db; no voters:
I20260620 21:59:19.140424 2839 leader_election.cc:288] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:19.140539 2844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:19.140776 2844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [term 1 LEADER]: Becoming Leader. State: Replica: 2874af00571e4e1bb5fc4eeade10e9db, State: Running, Role: LEADER
I20260620 21:59:19.141070 2844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } }
I20260620 21:59:19.141776 2839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:19.142877 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2874af00571e4e1bb5fc4eeade10e9db. Latest consensus state: current_term: 1 leader_uuid: "2874af00571e4e1bb5fc4eeade10e9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } } }
I20260620 21:59:19.142858 2845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2874af00571e4e1bb5fc4eeade10e9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2874af00571e4e1bb5fc4eeade10e9db" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33751 } } }
I20260620 21:59:19.143002 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:19.143002 2845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:19.143539 2853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:19.146549 2853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:19.151870 2853 catalog_manager.cc:1347] Generated new cluster ID: 670ef58194c24c42827931e83db00e96
I20260620 21:59:19.151942 2853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:19.167240 2853 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:19.168576 2853 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:19.175740 2853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: Generated new TSK 0
I20260620 21:59:19.176481 2853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:19.181104 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--builtin_ntp_servers=127.17.67.212:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:19.292899 2863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:19.293207 2863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:19.293304 2863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:19.297372 2863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:19.297428 2863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:19.297515 2863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:19.302304 2863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:19.303368 2863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:19.304738 2863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:19.314224 2868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.314210 2871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.314226 2869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.314589 2863 server_base.cc:1034] running on GCE node
I20260620 21:59:19.314996 2863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:19.315681 2863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:19.316923 2863 hybrid_clock.cc:648] HybridClock initialized: now 1781992759316905 us; error 26 us; skew 500 ppm
I20260620 21:59:19.318881 2863 webserver.cc:460] Webserver started at http://127.17.67.193:44983/ using document root <none> and password file <none>
I20260620 21:59:19.319550 2863 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:19.319653 2863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:19.319903 2863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:19.322141 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "1dbf17bb70bb4d209fee70d545ccbc08"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "f6cb63114cb3bfe2ef61cd6fe1a27a0e"
server_key_iv: "23bac2c0daf7b92c49187fb7c52a538a"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.322748 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "1dbf17bb70bb4d209fee70d545ccbc08"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "f6cb63114cb3bfe2ef61cd6fe1a27a0e"
server_key_iv: "23bac2c0daf7b92c49187fb7c52a538a"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.326591 2863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:59:19.329092 2877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.330444 2863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:19.330545 2863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "1dbf17bb70bb4d209fee70d545ccbc08"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "f6cb63114cb3bfe2ef61cd6fe1a27a0e"
server_key_iv: "23bac2c0daf7b92c49187fb7c52a538a"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.330636 2863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:19.355594 2863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:19.356284 2863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:19.356462 2863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:19.357034 2863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:19.358101 2863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:19.358152 2863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.358194 2863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:19.358211 2863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.367763 2863 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:35945
I20260620 21:59:19.367790 2990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:35945 every 8 connection(s)
I20260620 21:59:19.368772 2863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:19.377703 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2863
I20260620 21:59:19.377876 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:19.377969 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.370599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:35933 (local address 127.17.67.254:33751)
0620 21:59:19.370765 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.370770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.371802 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 21:59:19.373259 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.373261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.373330 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.373426 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:19.374384 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.374931 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.376068 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.376282 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.376391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.377428 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:19.377431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:19.377434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:19.377453 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:19.377471 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:19.377479 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:19.377589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.377745 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.377826 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:59:19.378275 17679 external_mini_cluster.cc:1428] Setting key dce1493b669995c8c54be745cb885024
I20260620 21:59:19.378842 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.370985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33751 (local address 127.17.67.193:35933)
0620 21:59:19.371650 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.371694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.373069 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.373673 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.373679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.373761 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:19.374239 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.374251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.375074 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.375078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.375931 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.375940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.376430 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.376434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.376598 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.377180 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:19.377182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:19.377218 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:19.377277 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:19.377594 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:19.377604 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:19.377612 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.377745 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.378002 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260620 21:59:19.380069 2991 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33751
I20260620 21:59:19.380231 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--builtin_ntp_servers=127.17.67.212:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:19.380324 2991 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:19.380882 2991 heartbeater.cc:511] Master 127.17.67.254:33751 requested a full tablet report, sending...
I20260620 21:59:19.382412 2802 ts_manager.cc:194] Registered new tserver with Master: 1dbf17bb70bb4d209fee70d545ccbc08 (127.17.67.193:35945)
I20260620 21:59:19.384240 2802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.193:35933
W20260620 21:59:19.495698 2994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:19.495949 2994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:19.495985 2994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:19.499852 2994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:19.499898 2994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:19.499950 2994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:19.504474 2994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:19.505412 2994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:19.506701 2994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:19.515998 3000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.515995 2999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.516244 2994 server_base.cc:1034] running on GCE node
W20260620 21:59:19.516393 3002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.516759 2994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:19.517372 2994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:19.518524 2994 hybrid_clock.cc:648] HybridClock initialized: now 1781992759518512 us; error 65 us; skew 500 ppm
I20260620 21:59:19.520329 2994 webserver.cc:460] Webserver started at http://127.17.67.194:38193/ using document root <none> and password file <none>
I20260620 21:59:19.520876 2994 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:19.520934 2994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:19.521134 2994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:19.523137 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "db86c03df451488ab2ff489ed861eef6"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "5ae83741876683bdeaab3f7d5dbc0c9b"
server_key_iv: "ff52140389ed665befddc5ff3d96ec6d"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.523680 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "db86c03df451488ab2ff489ed861eef6"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "5ae83741876683bdeaab3f7d5dbc0c9b"
server_key_iv: "ff52140389ed665befddc5ff3d96ec6d"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.527217 2994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:19.529618 3008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.530884 2994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:19.530989 2994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "db86c03df451488ab2ff489ed861eef6"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "5ae83741876683bdeaab3f7d5dbc0c9b"
server_key_iv: "ff52140389ed665befddc5ff3d96ec6d"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.531117 2994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:19.555060 2994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:19.555868 2994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:19.556085 2994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:19.556717 2994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:19.557693 2994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:19.557740 2994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.557812 2994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:19.557853 2994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.566603 2994 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:33079
I20260620 21:59:19.566639 3121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:33079 every 8 connection(s)
I20260620 21:59:19.567497 2994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:19.576668 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.569451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:48055 (local address 127.17.67.254:33751)
0620 21:59:19.569641 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.569646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.570698 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 21:59:19.572073 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.572076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.572124 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.572201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:19.573091 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.573677 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.574848 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.575070 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.575179 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.576189 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:19.576193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:19.576195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:19.576214 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:19.576230 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:19.576238 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:19.576311 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.576468 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.576524 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:59:19.577008 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 2994
I20260620 21:59:19.577145 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:19.577445 17679 external_mini_cluster.cc:1428] Setting key 70c21d6bad4ca997c0811557779626b1
I20260620 21:59:19.577698 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.569887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33751 (local address 127.17.67.194:48055)
0620 21:59:19.570541 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.570582 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.571866 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.572362 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.572368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.572449 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:19.572947 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.572959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.573829 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.573834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.574718 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.574727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.575216 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.575220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.575378 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.575944 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:19.575945 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:19.575983 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:19.576042 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:19.576324 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:19.576334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:19.576342 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.576467 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.576759 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260620 21:59:19.578825 3122 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33751
I20260620 21:59:19.579054 3122 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:19.579576 3122 heartbeater.cc:511] Master 127.17.67.254:33751 requested a full tablet report, sending...
I20260620 21:59:19.579711 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--builtin_ntp_servers=127.17.67.212:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:19.580647 2802 ts_manager.cc:194] Registered new tserver with Master: db86c03df451488ab2ff489ed861eef6 (127.17.67.194:33079)
I20260620 21:59:19.581362 2802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.194:48055
W20260620 21:59:19.689291 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:19.689651 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:19.689739 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:19.694288 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:19.694365 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:19.694458 3125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:19.699147 3125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:19.700205 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:19.701663 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:19.711529 3125 server_base.cc:1034] running on GCE node
W20260620 21:59:19.711472 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.711479 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:19.711476 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:19.712157 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:19.712766 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:19.713946 3125 hybrid_clock.cc:648] HybridClock initialized: now 1781992759713929 us; error 38 us; skew 500 ppm
I20260620 21:59:19.715791 3125 webserver.cc:460] Webserver started at http://127.17.67.195:39847/ using document root <none> and password file <none>
I20260620 21:59:19.716377 3125 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:19.716468 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:19.716701 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:19.718804 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "287c93c6ad274c8385329c7c12327cf4"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "e1d105ec0040ebe3994bb934460743ed"
server_key_iv: "e79dab1d9f3f52460397df78383c70ac"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.719422 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "287c93c6ad274c8385329c7c12327cf4"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "e1d105ec0040ebe3994bb934460743ed"
server_key_iv: "e79dab1d9f3f52460397df78383c70ac"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.722991 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:19.725531 3139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.726891 3125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:19.726992 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "287c93c6ad274c8385329c7c12327cf4"
format_stamp: "Formatted at 2026-06-20 21:59:19 on dist-test-slave-cx8d"
server_key: "e1d105ec0040ebe3994bb934460743ed"
server_key_iv: "e79dab1d9f3f52460397df78383c70ac"
server_key_version: "encryptionkey@0"
I20260620 21:59:19.727082 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:19.751339 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:19.752044 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:19.752206 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:19.752777 3125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:19.753811 3125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:19.753865 3125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.753913 3125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:19.753940 3125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:19.763602 3125 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:33685
I20260620 21:59:19.763644 3252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:33685 every 8 connection(s)
I20260620 21:59:19.764581 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:19.766065 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3125
I20260620 21:59:19.766193 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:19.766541 17679 external_mini_cluster.cc:1428] Setting key cbfb2fc62a6ac1c9b361931e6c2d69c7
I20260620 21:59:19.774469 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.766559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:57435 (local address 127.17.67.254:33751)
0620 21:59:19.766725 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.766729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.768019 (+ 1290us) server_negotiation.cc:407] Connection header received
0620 21:59:19.769690 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.769693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.769739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.769807 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:19.770736 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.771247 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.772451 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.772668 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.772748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.773883 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:19.773887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:19.773889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:19.773902 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:19.773923 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:19.773932 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:19.774009 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.774230 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.774302 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 21:59:19.775275 3255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.767115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33751 (local address 127.17.67.195:57435)
0620 21:59:19.767877 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.767933 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.769434 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.769992 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.769999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.770090 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:19.770614 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.770628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.771364 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.771368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.772328 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.772337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.772796 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.772799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.772955 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.773614 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:19.773617 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:19.773667 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:19.773733 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:19.774022 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:19.774029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:19.774033 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.774164 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.774456 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":155,"threads_started":1}
I20260620 21:59:19.776373 3253 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33751
I20260620 21:59:19.776621 3253 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:19.777146 3253 heartbeater.cc:511] Master 127.17.67.254:33751 requested a full tablet report, sending...
I20260620 21:59:19.778182 2802 ts_manager.cc:194] Registered new tserver with Master: 287c93c6ad274c8385329c7c12327cf4 (127.17.67.195:33685)
I20260620 21:59:19.778740 2802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.195:57435
I20260620 21:59:19.780532 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:19.786644 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.782497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.17.67.254:33751)
0620 21:59:19.782656 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.782660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.782815 (+ 155us) server_negotiation.cc:407] Connection header received
0620 21:59:19.782964 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.782967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.783009 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.783115 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:19.783399 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.784241 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.785149 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.785354 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.785431 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.786247 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:19.786250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:19.786252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:19.786263 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:19.786280 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:19.786287 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:19.786343 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.786433 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.786477 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:59:19.789340 2802 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:19.792357 2802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:19.806581 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.801169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.17.67.193:35945)
0620 21:59:19.802022 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.802036 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.802058 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:19.802139 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.802143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.802324 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.802507 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:19.802768 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.803895 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805251 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.805844 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805968 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806189 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.806304 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.806410 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":106,"threads_started":1}
I20260620 21:59:19.806576 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.800405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:35945 (local address 127.0.0.1:34674)
0620 21:59:19.801318 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.801331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.801430 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.802495 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.802497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.802509 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:19.802654 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.802661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.804019 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.804023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.805143 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.805149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.805965 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.806135 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806158 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.806275 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.806413 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":136,"thread_start_us":120,"threads_started":1}
I20260620 21:59:19.807029 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.801246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.17.67.195:33685)
0620 21:59:19.801914 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.801924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.801944 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:59:19.802005 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.802009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.802149 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.802304 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:19.802592 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.803467 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805022 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.805843 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.806398 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806570 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.806621 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.806863 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":170,"threads_started":1}
I20260620 21:59:19.807044 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.800321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:33079 (local address 127.0.0.1:49620)
0620 21:59:19.800763 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.800809 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.800930 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.802473 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.802479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.802499 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:19.802835 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.802841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.804253 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.804256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.805363 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.805374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.806634 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.806638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.806744 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806773 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.806851 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.806916 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":219,"thread_start_us":107,"threads_started":1}
I20260620 21:59:19.807168 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.800437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.17.67.194:33079)
0620 21:59:19.801787 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:19.801801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:19.801826 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:59:19.801902 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:19.801909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:19.802133 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:19.802303 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:19.802955 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.804124 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805512 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.806304 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.806397 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806570 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:19.806905 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 21:59:19.806999 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":114,"threads_started":1}
I20260620 21:59:19.808211 3265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:19.801083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33685 (local address 127.0.0.1:38640)
0620 21:59:19.801553 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:19.801587 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:19.801679 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:19.802301 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:19.802308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:19.802329 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:19.802495 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.802503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.803588 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.803594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.804888 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:19.804899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:19.805989 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:19.805993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:19.806103 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:19.806153 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 21:59:19.806244 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:59:19.808088 (+ 1844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":1721,"thread_start_us":113,"threads_started":1}
I20260620 21:59:19.809983 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa7d30962a41beaa12fee454e86280 (DEFAULT_TABLE table=test-table [id=77153764a1f743749f5e08906679f544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:19.810024 2925 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa7d30962a41beaa12fee454e86280 (DEFAULT_TABLE table=test-table [id=77153764a1f743749f5e08906679f544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:19.810861 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa7d30962a41beaa12fee454e86280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:19.811025 2925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa7d30962a41beaa12fee454e86280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:19.810725 3187 tablet_service.cc:1467] Processing CreateTablet for tablet 09aa7d30962a41beaa12fee454e86280 (DEFAULT_TABLE table=test-table [id=77153764a1f743749f5e08906679f544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:19.811596 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09aa7d30962a41beaa12fee454e86280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:19.816555 3270 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Bootstrap starting.
I20260620 21:59:19.818044 3272 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Bootstrap starting.
I20260620 21:59:19.819144 3270 tablet_bootstrap.cc:654] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:19.819983 3270 log.cc:824] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:19.820137 3272 tablet_bootstrap.cc:654] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:19.820735 3271 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Bootstrap starting.
I20260620 21:59:19.820935 3272 log.cc:824] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:19.821993 3270 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: No bootstrap required, opened a new log
I20260620 21:59:19.822201 3270 ts_tablet_manager.cc:1397] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:19.822768 3271 tablet_bootstrap.cc:654] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:19.823421 3272 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: No bootstrap required, opened a new log
I20260620 21:59:19.823604 3272 ts_tablet_manager.cc:1397] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 21:59:19.823624 3271 log.cc:824] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:19.825456 3271 tablet_bootstrap.cc:492] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: No bootstrap required, opened a new log
I20260620 21:59:19.825949 3271 ts_tablet_manager.cc:1397] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:59:19.826257 3270 raft_consensus.cc:348] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.826557 3270 raft_consensus.cc:374] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:19.826611 3270 raft_consensus.cc:729] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db86c03df451488ab2ff489ed861eef6, State: Initialized, Role: FOLLOWER
I20260620 21:59:19.826673 3272 raft_consensus.cc:348] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.826901 3272 raft_consensus.cc:374] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:19.826954 3272 raft_consensus.cc:729] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf17bb70bb4d209fee70d545ccbc08, State: Initialized, Role: FOLLOWER
I20260620 21:59:19.827060 3270 consensus_queue.cc:260] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.827303 3272 consensus_queue.cc:260] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.827723 3122 heartbeater.cc:503] Master 127.17.67.254:33751 was elected leader, sending a full tablet report...
I20260620 21:59:19.828012 3270 ts_tablet_manager.cc:1428] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:59:19.828217 3272 ts_tablet_manager.cc:1428] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260620 21:59:19.828574 2991 heartbeater.cc:503] Master 127.17.67.254:33751 was elected leader, sending a full tablet report...
I20260620 21:59:19.830684 3271 raft_consensus.cc:348] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.830922 3271 raft_consensus.cc:374] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:19.830986 3271 raft_consensus.cc:729] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287c93c6ad274c8385329c7c12327cf4, State: Initialized, Role: FOLLOWER
I20260620 21:59:19.831444 3271 consensus_queue.cc:260] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:19.832091 3253 heartbeater.cc:503] Master 127.17.67.254:33751 was elected leader, sending a full tablet report...
I20260620 21:59:19.832334 3271 ts_tablet_manager.cc:1428] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 21:59:19.870100 2992 tablet.cc:2369] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:20.008968 3276 raft_consensus.cc:482] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:20.009243 3276 raft_consensus.cc:504] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:20.010532 3276 leader_election.cc:288] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf17bb70bb4d209fee70d545ccbc08 (127.17.67.193:35945), 287c93c6ad274c8385329c7c12327cf4 (127.17.67.195:33685)
I20260620 21:59:20.014183 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.010741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:35945 (local address 127.17.67.194:52863)
0620 21:59:20.010905 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:20.010918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:20.011023 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:20.011367 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:20.011370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:20.011387 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:20.011505 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.011512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.012269 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.012273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.013061 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.013069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.013751 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.013755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.013857 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.013883 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:59:20.013969 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:59:20.014042 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 21:59:20.014183 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.010969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:52863 (local address 127.17.67.193:35945)
0620 21:59:20.011138 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.011142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.011158 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:59:20.011217 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.011220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.011273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.011354 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:20.011652 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.012156 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.013180 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.013624 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.013764 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.013819 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.013986 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.014073 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:59:20.014596 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.011247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58147 (local address 127.17.67.195:33685)
0620 21:59:20.011413 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.011417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.011432 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:59:20.011603 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.011606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.011656 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.011738 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:20.012029 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.012514 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.013513 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.013979 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.014118 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.014163 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.014395 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.014469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:59:20.014919 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.010946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:33685 (local address 127.17.67.194:58147)
0620 21:59:20.011319 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:20.011335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:20.011442 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:20.011762 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:20.011764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:20.011780 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:20.011893 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.011898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.012623 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.012630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.013382 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:20.013391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.014110 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.014113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:20.014211 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.014258 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 21:59:20.014736 (+ 478us) client_negotiation.cc:240] Negotiation successful
0620 21:59:20.014817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":148,"threads_started":1}
I20260620 21:59:20.014914 2945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa7d30962a41beaa12fee454e86280" candidate_uuid: "db86c03df451488ab2ff489ed861eef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" is_pre_election: true
I20260620 21:59:20.015264 2945 raft_consensus.cc:2454] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db86c03df451488ab2ff489ed861eef6 in term 0.
W20260620 21:59:20.015476 3254 tablet.cc:2369] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:20.015534 3207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa7d30962a41beaa12fee454e86280" candidate_uuid: "db86c03df451488ab2ff489ed861eef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287c93c6ad274c8385329c7c12327cf4" is_pre_election: true
I20260620 21:59:20.015765 3207 raft_consensus.cc:2454] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db86c03df451488ab2ff489ed861eef6 in term 0.
I20260620 21:59:20.015806 3009 leader_election.cc:302] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf17bb70bb4d209fee70d545ccbc08, db86c03df451488ab2ff489ed861eef6; no voters:
I20260620 21:59:20.016157 3276 raft_consensus.cc:2781] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:20.016255 3276 raft_consensus.cc:482] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:20.016315 3276 raft_consensus.cc:3037] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:20.017313 3276 raft_consensus.cc:504] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:20.017714 3276 leader_election.cc:288] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf17bb70bb4d209fee70d545ccbc08 (127.17.67.193:35945), 287c93c6ad274c8385329c7c12327cf4 (127.17.67.195:33685)
I20260620 21:59:20.018067 2945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa7d30962a41beaa12fee454e86280" candidate_uuid: "db86c03df451488ab2ff489ed861eef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf17bb70bb4d209fee70d545ccbc08"
I20260620 21:59:20.018085 3207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09aa7d30962a41beaa12fee454e86280" candidate_uuid: "db86c03df451488ab2ff489ed861eef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287c93c6ad274c8385329c7c12327cf4"
I20260620 21:59:20.018199 2945 raft_consensus.cc:3037] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:20.018211 3207 raft_consensus.cc:3037] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:20.019172 2945 raft_consensus.cc:2454] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db86c03df451488ab2ff489ed861eef6 in term 1.
I20260620 21:59:20.019260 3207 raft_consensus.cc:2454] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db86c03df451488ab2ff489ed861eef6 in term 1.
I20260620 21:59:20.019464 3009 leader_election.cc:302] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf17bb70bb4d209fee70d545ccbc08, db86c03df451488ab2ff489ed861eef6; no voters:
I20260620 21:59:20.019646 3276 raft_consensus.cc:2781] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:20.019877 3276 raft_consensus.cc:686] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 1 LEADER]: Becoming Leader. State: Replica: db86c03df451488ab2ff489ed861eef6, State: Running, Role: LEADER
I20260620 21:59:20.020210 3276 consensus_queue.cc:237] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } }
I20260620 21:59:20.023692 2803 catalog_manager.cc:5495] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 reported cstate change: term changed from 0 to 1, leader changed from <none> to db86c03df451488ab2ff489ed861eef6 (127.17.67.194). New cstate: current_term: 1 leader_uuid: "db86c03df451488ab2ff489ed861eef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db86c03df451488ab2ff489ed861eef6" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:20.044154 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.040170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.17.67.194:33079)
0620 21:59:20.040300 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:20.040305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:20.040319 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:20.040451 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:20.040455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:20.040530 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:20.040646 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:20.040886 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.041428 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.042438 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:20.042728 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:20.042839 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:20.043230 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:20.043725 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:20.043882 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:20.043943 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 21:59:20.043995 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:20.049945 2945 raft_consensus.cc:1261] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08 [term 1 FOLLOWER]: Refusing update from remote peer db86c03df451488ab2ff489ed861eef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:20.049963 3207 raft_consensus.cc:1261] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 1 FOLLOWER]: Refusing update from remote peer db86c03df451488ab2ff489ed861eef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:20.050697 3280 consensus_queue.cc:1035] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf17bb70bb4d209fee70d545ccbc08" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:20.050880 3276 consensus_queue.cc:1035] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287c93c6ad274c8385329c7c12327cf4" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:20.063308 3284 mvcc.cc:205] Tried to move back new op lower bound from 7299042345157001216 to 7299042345045221376. Current Snapshot: MvccSnapshot[applied={T|T < 7299042345157001216 or (T in {7299042345157001216})}]
I20260620 21:59:20.066339 2801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:20.066474 2801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:20.069113 2801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2874af00571e4e1bb5fc4eeade10e9db: Sending DeleteTablet for 3 replicas of tablet 09aa7d30962a41beaa12fee454e86280
I20260620 21:59:20.069921 2925 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa7d30962a41beaa12fee454e86280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:20 UTC) from {username='slave'} at 127.0.0.1:34674
I20260620 21:59:20.069902 3187 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa7d30962a41beaa12fee454e86280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:20 UTC) from {username='slave'} at 127.0.0.1:38640
I20260620 21:59:20.070407 3292 tablet_replica.cc:330] T 09aa7d30962a41beaa12fee454e86280 P 1dbf17bb70bb4d209fee70d545ccbc08: stopping tablet replica
I20260620 21:59:20.070395 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet 09aa7d30962a41beaa12fee454e86280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:20 UTC) from {username='slave'} at 127.0.0.1:49620
I20260620 21:59:20.070667 3293 tablet_replica.cc:330] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: stopping tablet replica
I20260620 21:59:20.070802 3294 tablet_replica.cc:330] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: stopping tablet replica
I20260620 21:59:20.070971 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2863
I20260620 21:59:20.071240 3294 raft_consensus.cc:2229] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:20.071369 3293 raft_consensus.cc:2229] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:20.071600 3294 raft_consensus.cc:2258] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:20.071802 3293 raft_consensus.cc:2258] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:20.073315 3294 ts_tablet_manager.cc:1905] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:20.073315 3293 ts_tablet_manager.cc:1905] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:20.076095 3293 ts_tablet_manager.cc:1918] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:20.076256 3293 log.cc:1196] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/09aa7d30962a41beaa12fee454e86280
I20260620 21:59:20.076618 3293 ts_tablet_manager.cc:1939] T 09aa7d30962a41beaa12fee454e86280 P 287c93c6ad274c8385329c7c12327cf4: Deleting consensus metadata
I20260620 21:59:20.076733 3294 ts_tablet_manager.cc:1918] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:20.076836 3294 log.cc:1196] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/09aa7d30962a41beaa12fee454e86280
I20260620 21:59:20.077163 3294 ts_tablet_manager.cc:1939] T 09aa7d30962a41beaa12fee454e86280 P db86c03df451488ab2ff489ed861eef6: Deleting consensus metadata
I20260620 21:59:20.078104 2789 catalog_manager.cc:4841] TS 287c93c6ad274c8385329c7c12327cf4 (127.17.67.195:33685): tablet 09aa7d30962a41beaa12fee454e86280 (table test-table [id=77153764a1f743749f5e08906679f544]) successfully deleted
I20260620 21:59:20.078418 2789 catalog_manager.cc:4841] TS db86c03df451488ab2ff489ed861eef6 (127.17.67.194:33079): tablet 09aa7d30962a41beaa12fee454e86280 (table test-table [id=77153764a1f743749f5e08906679f544]) successfully deleted
W20260620 21:59:20.080768 2787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:35945 (error 108)
I20260620 21:59:20.081633 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2994
I20260620 21:59:20.081863 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:20.081516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:35945 (local address 127.0.0.1:34682)
0620 21:59:20.081683 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:20.081763 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 21:59:20.082463 2787 catalog_manager.cc:4584] TS 1dbf17bb70bb4d209fee70d545ccbc08 (127.17.67.193:35945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09aa7d30962a41beaa12fee454e86280: Network error: Client connection negotiation failed: client connection to 127.17.67.193:35945: connect: Connection refused (error 111)
I20260620 21:59:20.087460 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3125
I20260620 21:59:20.092775 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 2772
2026-06-20T21:59:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:20 dist-test-slave-cx8d krb5kdc[3298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:20 dist-test-slave-cx8d krb5kdc[3298](info): set up 2 sockets
Jun 20 21:59:20 dist-test-slave-cx8d krb5kdc[3298](info): commencing operation
krb5kdc: starting...
W20260620 21:59:21.129452 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:21.250124 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:33909
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:21.358388 3311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.358716 3311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.358798 3311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.362818 3311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:21.362914 3311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.362953 3311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.363010 3311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:21.363049 3311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:21.368273 3311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:21.369346 3311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.370404 3311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.378968 3317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.378953 3316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.378953 3319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.379374 3311 server_base.cc:1034] running on GCE node
I20260620 21:59:21.380004 3311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.381126 3311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.382339 3311 hybrid_clock.cc:648] HybridClock initialized: now 1781992761382302 us; error 49 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.385150 3311 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:21.386130 3311 webserver.cc:460] Webserver started at http://127.17.67.254:46603/ using document root <none> and password file <none>
I20260620 21:59:21.386683 3311 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.386771 3311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.386998 3311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:21.389025 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "d4b07a64dd344735b3edbb062fb53100"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "c047e8a13bac13bf787d0e8408f5803a"
server_key_iv: "b4baa20e74f72ced4f4a594092deaa59"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.389652 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "d4b07a64dd344735b3edbb062fb53100"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "c047e8a13bac13bf787d0e8408f5803a"
server_key_iv: "b4baa20e74f72ced4f4a594092deaa59"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.393239 3311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:21.395644 3326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.397027 3311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:21.397193 3311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "d4b07a64dd344735b3edbb062fb53100"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "c047e8a13bac13bf787d0e8408f5803a"
server_key_iv: "b4baa20e74f72ced4f4a594092deaa59"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.397315 3311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.424093 3311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.431207 3311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.431423 3311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.438854 3311 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:33909
I20260620 21:59:21.438938 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:33909 every 8 connection(s)
I20260620 21:59:21.439875 3311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:21.442739 3379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:21.446450 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3311
I20260620 21:59:21.446565 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:21.446883 17679 external_mini_cluster.cc:1428] Setting key ea6dc28b11863995525724ae22dfaa10
I20260620 21:59:21.449215 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: Bootstrap starting.
I20260620 21:59:21.452020 3379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:21.452991 3379 log.cc:824] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:21.455328 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: No bootstrap required, opened a new log
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:21.459833 3379 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } }
I20260620 21:59:21.460145 3379 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:21.460216 3379 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b07a64dd344735b3edbb062fb53100, State: Initialized, Role: FOLLOWER
I20260620 21:59:21.460752 3379 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } }
I20260620 21:59:21.460942 3379 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:21.461037 3379 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:21.461164 3379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:21.462400 3379 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } }
I20260620 21:59:21.462775 3379 leader_election.cc:302] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b07a64dd344735b3edbb062fb53100; no voters:
I20260620 21:59:21.463124 3379 leader_election.cc:288] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:21.463236 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:21.463471 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [term 1 LEADER]: Becoming Leader. State: Replica: d4b07a64dd344735b3edbb062fb53100, State: Running, Role: LEADER
I20260620 21:59:21.463779 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.448446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.17.67.254:33909)
0620 21:59:21.448882 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.448902 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.448946 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:59:21.449887 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.449911 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.450342 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.450753 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.451818 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.453639 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.454605 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.455314 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.455503 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.458106 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:21.458116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:21.458141 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:21.458186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:21.460314 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.460738 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.460744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.460757 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.460911 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.461306 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.461309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.461311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.461882 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:21.462100 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:21.462297 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 21:59:21.462638 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":101,"threads_started":1}
I20260620 21:59:21.464462 3379 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:21.463771 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } }
I20260620 21:59:21.466866 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b07a64dd344735b3edbb062fb53100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } } }
I20260620 21:59:21.467061 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b07a64dd344735b3edbb062fb53100. Latest consensus state: current_term: 1 leader_uuid: "d4b07a64dd344735b3edbb062fb53100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b07a64dd344735b3edbb062fb53100" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 33909 } } }
I20260620 21:59:21.467338 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:21.467345 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:21.469908 3398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:21.469990 3398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:21.470052 3399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:21.470983 3399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:21.476939 3399 catalog_manager.cc:1347] Generated new cluster ID: f7b7741f029945f38c07bd3a791baac0
I20260620 21:59:21.477015 3399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:21.514817 3399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:21.516141 3399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:21.524199 3399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: Generated new TSK 0
I20260620 21:59:21.525015 3399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:59:21.561125 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:21.678732 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.679045 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.679150 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.683748 3405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.683840 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.683945 3405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:21.689931 3405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:21.691152 3405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.692207 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.702157 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.702196 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.702365 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.702378 3405 server_base.cc:1034] running on GCE node
I20260620 21:59:21.702977 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.703604 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.704788 3405 hybrid_clock.cc:648] HybridClock initialized: now 1781992761704745 us; error 53 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.707726 3405 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:59:21.708613 3405 webserver.cc:460] Webserver started at http://127.17.67.193:43717/ using document root <none> and password file <none>
I20260620 21:59:21.709154 3405 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.709213 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.709416 3405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:21.711406 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "91b085dd32de420698281cdaf27b136b"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "aac6de34fbf71d4fa04b4852f4e09d0a"
server_key_iv: "6ee562f64d21332c8aba174214e8ea8b"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.711962 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "91b085dd32de420698281cdaf27b136b"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "aac6de34fbf71d4fa04b4852f4e09d0a"
server_key_iv: "6ee562f64d21332c8aba174214e8ea8b"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.715708 3405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:21.718176 3420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.719379 3405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:21.719473 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "91b085dd32de420698281cdaf27b136b"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "aac6de34fbf71d4fa04b4852f4e09d0a"
server_key_iv: "6ee562f64d21332c8aba174214e8ea8b"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.719631 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.737149 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.740460 3405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.740679 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.741292 3405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.742219 3405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.742290 3405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.742354 3405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.742399 3405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.751855 3405 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:40929
I20260620 21:59:21.751928 3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:40929 every 8 connection(s)
I20260620 21:59:21.752789 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:21.758733 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3405
I20260620 21:59:21.758925 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:21.759296 17679 external_mini_cluster.cc:1428] Setting key 80ecf41ed1dd37658a616278decab720
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:21.768214 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.754870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:36713 (local address 127.17.67.254:33909)
0620 21:59:21.755023 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.755027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.755898 (+ 871us) server_negotiation.cc:407] Connection header received
0620 21:59:21.757231 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.757234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.757286 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.757367 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.758672 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.759626 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.761204 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.761680 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.761811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.764469 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:21.764473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:21.764476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:21.764508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:21.766100 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.766686 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.766690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.766692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.766773 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.767117 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.767120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.767122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.767312 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:21.767416 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:21.767917 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:59:21.768050 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:59:21.769132 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.755295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33909 (local address 127.17.67.193:36713)
0620 21:59:21.755741 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.755783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.757011 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.757524 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.757558 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.757982 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.758457 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.758471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.759779 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.759784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.761069 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.761079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.761866 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.761871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.762093 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.762806 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.762833 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.764306 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:21.766252 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.766258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.766275 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.766582 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.766865 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.766869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.766870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.767014 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.767414 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:21.767421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:21.767685 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 21:59:21.767878 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 21:59:21.768220 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260620 21:59:21.770453 3534 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33909
I20260620 21:59:21.770716 3534 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
I20260620 21:59:21.771322 3534 heartbeater.cc:511] Master 127.17.67.254:33909 requested a full tablet report, sending...
I20260620 21:59:21.772951 3343 ts_manager.cc:194] Registered new tserver with Master: 91b085dd32de420698281cdaf27b136b (127.17.67.193:40929)
I20260620 21:59:21.774328 3343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:36713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:59:21.785974 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:21.898690 3539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:21.898936 3539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:21.899011 3539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:21.903211 3539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:21.903266 3539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:21.903357 3539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:21.908788 3539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:21.909962 3539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:21.910992 3539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:21.920411 3547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.920423 3544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:21.920428 3545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:21.920729 3539 server_base.cc:1034] running on GCE node
I20260620 21:59:21.921114 3539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:21.921864 3539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:21.923049 3539 hybrid_clock.cc:648] HybridClock initialized: now 1781992761923021 us; error 46 us; skew 500 ppm
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:21.925606 3539 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:59:21.926424 3539 webserver.cc:460] Webserver started at http://127.17.67.194:39563/ using document root <none> and password file <none>
I20260620 21:59:21.926963 3539 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:21.927062 3539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:21.927286 3539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:21.929225 3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "dc945746fb14452391fe31f341722244"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "388557ab5b3f7b1a7c04c45439d6a892"
server_key_iv: "a4fa260bdf082033f3ab7cebdff5dcfc"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.929809 3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "dc945746fb14452391fe31f341722244"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "388557ab5b3f7b1a7c04c45439d6a892"
server_key_iv: "a4fa260bdf082033f3ab7cebdff5dcfc"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.933307 3539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:21.935765 3554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.937117 3539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:21.937275 3539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "dc945746fb14452391fe31f341722244"
format_stamp: "Formatted at 2026-06-20 21:59:21 on dist-test-slave-cx8d"
server_key: "388557ab5b3f7b1a7c04c45439d6a892"
server_key_iv: "a4fa260bdf082033f3ab7cebdff5dcfc"
server_key_version: "encryptionkey@0"
I20260620 21:59:21.937397 3539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:21.955188 3539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:21.958405 3539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:21.958644 3539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:21.959282 3539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:21.960198 3539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:21.960276 3539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.960354 3539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:21.960421 3539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:21.971870 3539 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:44107
I20260620 21:59:21.971886 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:44107 every 8 connection(s)
I20260620 21:59:21.972886 3539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:21.982323 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3539
I20260620 21:59:21.982430 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:21.982735 17679 external_mini_cluster.cc:1428] Setting key 12af7d8171155130562eee7e13fc82b8
Jun 20 21:59:21 dist-test-slave-cx8d krb5kdc[3298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992761, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:21.987812 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.974833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:45137 (local address 127.17.67.254:33909)
0620 21:59:21.974969 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:21.974972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:21.975796 (+ 824us) server_negotiation.cc:407] Connection header received
0620 21:59:21.977177 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:21.977181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:21.977240 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:21.977425 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:21.978706 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.979254 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.980657 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.980894 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.981006 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.983759 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:21.983762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:21.983764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:21.983788 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:21.985237 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.985911 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.985913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.985915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.986067 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:21.986444 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:21.986449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:21.986451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:21.986706 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:21.986815 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:21.987423 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 21:59:21.987569 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:59:21.988605 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:21.975196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33909 (local address 127.17.67.194:45137)
0620 21:59:21.975643 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:21.975685 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:21.976966 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:21.977648 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:21.977655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:21.978022 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:21.978556 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.978570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.979393 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.979398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.980530 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:21.980541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:21.981047 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:21.981051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:21.981230 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:21.981906 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:21.981923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:21.983603 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:21.985359 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.985368 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.985387 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.985804 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.986183 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:21.986186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:21.986188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:21.986341 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:21.986884 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:21.986893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:21.987157 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 21:59:21.987319 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:59:21.987616 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260620 21:59:21.990159 3668 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33909
I20260620 21:59:21.990442 3668 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:21.990971 3668 heartbeater.cc:511] Master 127.17.67.254:33909 requested a full tablet report, sending...
I20260620 21:59:21.992062 3343 ts_manager.cc:194] Registered new tserver with Master: dc945746fb14452391fe31f341722244 (127.17.67.194:44107)
I20260620 21:59:21.992664 3343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:45137
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:59:22.010862 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:22.130177 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:22.130463 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:22.130508 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:22.134719 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:22.134802 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:22.134865 3673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:22.140719 3673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:22.141880 3673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:22.142921 3673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:22.152489 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:22.152463 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:22.152463 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:22.152782 3673 server_base.cc:1034] running on GCE node
I20260620 21:59:22.153329 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:22.154033 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:22.155225 3673 hybrid_clock.cc:648] HybridClock initialized: now 1781992762155188 us; error 50 us; skew 500 ppm
Jun 20 21:59:22 dist-test-slave-cx8d krb5kdc[3298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992762, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:22.158247 3673 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:59:22.159148 3673 webserver.cc:460] Webserver started at http://127.17.67.195:39713/ using document root <none> and password file <none>
I20260620 21:59:22.159754 3673 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:22.159821 3673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:22.160044 3673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:22.162206 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "337b2bdee6e344ea9f8c630744ae1f5f"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "244673e153fdb9f6e411b55d09c715cd"
server_key_iv: "fdc25304f03e5b3bb5ed4d977b666472"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.162796 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "337b2bdee6e344ea9f8c630744ae1f5f"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "244673e153fdb9f6e411b55d09c715cd"
server_key_iv: "fdc25304f03e5b3bb5ed4d977b666472"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.167138 3673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:59:22.169669 3688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.171034 3673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:22.171171 3673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "337b2bdee6e344ea9f8c630744ae1f5f"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "244673e153fdb9f6e411b55d09c715cd"
server_key_iv: "fdc25304f03e5b3bb5ed4d977b666472"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.171290 3673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:22.189853 3673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:22.192865 3673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:22.193065 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:22.193686 3673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:22.194509 3673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:22.194564 3673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.194631 3673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:22.194669 3673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.204460 3673 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:40581
I20260620 21:59:22.204488 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:40581 every 8 connection(s)
I20260620 21:59:22.205334 3673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:22.207645 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3673
I20260620 21:59:22.207809 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:22.208178 17679 external_mini_cluster.cc:1428] Setting key 0e6c59cb79d793dcce3b9f7723ed3fe7
Jun 20 21:59:22 dist-test-slave-cx8d krb5kdc[3298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992762, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:22.219676 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.207157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:60621 (local address 127.17.67.254:33909)
0620 21:59:22.207310 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.207314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.208284 (+ 970us) server_negotiation.cc:407] Connection header received
0620 21:59:22.210105 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.210108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.210164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.210244 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.211451 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.211986 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.213157 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.213370 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.213543 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.216054 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:22.216063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:22.216066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:22.216089 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:22.217676 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:22.218181 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:22.218186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:22.218188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:22.218268 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:22.218616 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:22.218620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:22.218622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:22.218804 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:22.218919 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.219392 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.219514 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:59:22.220505 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.207600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:33909 (local address 127.17.67.195:60621)
0620 21:59:22.208114 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.208174 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.209891 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.210465 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.210473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.210827 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:22.211308 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.211323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.212105 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.212108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.213029 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.213038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.213628 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.213633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.213776 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.214334 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:22.214350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:22.215891 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:22.217799 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:22.217805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:22.217820 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:22.218080 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:22.218361 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:22.218366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:22.218368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:22.218514 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:22.218908 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:22.218914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:22.219165 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.219353 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.219680 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260620 21:59:22.221797 3802 heartbeater.cc:347] Connected to a master server at 127.17.67.254:33909
I20260620 21:59:22.222018 3802 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:22.222536 3802 heartbeater.cc:511] Master 127.17.67.254:33909 requested a full tablet report, sending...
I20260620 21:59:22.223536 3343 ts_manager.cc:194] Registered new tserver with Master: 337b2bdee6e344ea9f8c630744ae1f5f (127.17.67.195:40581)
I20260620 21:59:22.224133 3343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:60621
I20260620 21:59:22.234490 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:22.246189 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.236550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.17.67.254:33909)
0620 21:59:22.236687 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.236692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.236836 (+ 144us) server_negotiation.cc:407] Connection header received
0620 21:59:22.236998 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.237001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.237046 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.237139 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.237988 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.238708 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.239744 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.239995 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.240134 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.242107 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:22.242111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:22.242113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:22.242143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:22.243727 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:22.244327 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:22.244339 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:22.244341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:22.244425 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:22.244798 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:22.244800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:22.244801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:22.244992 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:22.245152 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.245801 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.245879 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:59:22.249945 3343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:22.252895 3343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:22.267778 3815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.261725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.17.67.194:44107)
0620 21:59:22.262708 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.262719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.262736 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:59:22.262797 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.262801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.262938 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.263081 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:22.263471 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264685 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.266423 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267093 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267223 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267377 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.267525 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.267626 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":106,"threads_started":1}
I20260620 21:59:22.267772 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.261486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:40581 (local address 127.0.0.1:56206)
0620 21:59:22.262107 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.262121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.262212 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.263092 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.263098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.263120 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:22.263245 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.263252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264594 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.264600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.266272 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.266278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267354 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.267444 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267466 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.267519 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.267630 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":109,"thread_start_us":122,"threads_started":1}
I20260620 21:59:22.268045 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.262304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.17.67.193:40929)
0620 21:59:22.263137 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.263151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.263171 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:59:22.263239 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.263242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.263383 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.263569 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:22.263782 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264847 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.266328 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267212 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267330 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267462 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.267843 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.267923 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":76,"threads_started":1}
I20260620 21:59:22.268023 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.262207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.17.67.195:40581)
0620 21:59:22.262581 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.262594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.262617 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:59:22.262693 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.262698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.262899 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.263402 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:22.263565 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264472 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.266420 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267225 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267607 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267749 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.267798 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.267887 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":185,"threads_started":1}
I20260620 21:59:22.268436 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.262152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40929 (local address 127.0.0.1:45314)
0620 21:59:22.262820 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.262833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.262918 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.263550 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.263553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.263570 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:22.263661 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.263666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264979 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.264984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.266211 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.266220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267519 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.267623 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267641 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.267704 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.267773 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":50,"thread_start_us":84,"threads_started":1}
I20260620 21:59:22.268731 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.261381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:44107 (local address 127.0.0.1:43338)
0620 21:59:22.261718 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.261761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.261878 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.263239 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.263242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.263256 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:22.263344 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.263349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.264814 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.264817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.266264 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.266275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.267222 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.267225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.267334 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.267369 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.267483 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.268624 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":171,"thread_start_us":86,"threads_started":1}
I20260620 21:59:22.271437 3468 tablet_service.cc:1467] Processing CreateTablet for tablet 6577d67ba72c40e48c8d2351a035377e (DEFAULT_TABLE table=test-table [id=fdb07775c24d408187b3dd2019983642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:22.271440 3736 tablet_service.cc:1467] Processing CreateTablet for tablet 6577d67ba72c40e48c8d2351a035377e (DEFAULT_TABLE table=test-table [id=fdb07775c24d408187b3dd2019983642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:22.271679 3602 tablet_service.cc:1467] Processing CreateTablet for tablet 6577d67ba72c40e48c8d2351a035377e (DEFAULT_TABLE table=test-table [id=fdb07775c24d408187b3dd2019983642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:22.272545 3736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6577d67ba72c40e48c8d2351a035377e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:22.272541 3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6577d67ba72c40e48c8d2351a035377e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:22.272616 3602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6577d67ba72c40e48c8d2351a035377e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:22.279088 3819 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Bootstrap starting.
I20260620 21:59:22.280066 3820 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: Bootstrap starting.
I20260620 21:59:22.281450 3819 tablet_bootstrap.cc:654] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:22.281948 3820 tablet_bootstrap.cc:654] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:22.281955 3821 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Bootstrap starting.
I20260620 21:59:22.282658 3820 log.cc:824] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:22.282842 3819 log.cc:824] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:22.284010 3821 tablet_bootstrap.cc:654] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:22.284462 3820 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: No bootstrap required, opened a new log
I20260620 21:59:22.284596 3820 ts_tablet_manager.cc:1397] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:59:22.284888 3819 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: No bootstrap required, opened a new log
I20260620 21:59:22.285058 3819 ts_tablet_manager.cc:1397] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:22.285229 3821 log.cc:824] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:22.288280 3821 tablet_bootstrap.cc:492] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: No bootstrap required, opened a new log
I20260620 21:59:22.288542 3821 ts_tablet_manager.cc:1397] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:59:22.288918 3820 raft_consensus.cc:348] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.289134 3820 raft_consensus.cc:374] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:22.289173 3820 raft_consensus.cc:729] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b085dd32de420698281cdaf27b136b, State: Initialized, Role: FOLLOWER
I20260620 21:59:22.289112 3819 raft_consensus.cc:348] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.289348 3819 raft_consensus.cc:374] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:22.289397 3819 raft_consensus.cc:729] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc945746fb14452391fe31f341722244, State: Initialized, Role: FOLLOWER
I20260620 21:59:22.289616 3820 consensus_queue.cc:260] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.289826 3819 consensus_queue.cc:260] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.290643 3820 ts_tablet_manager.cc:1428] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:59:22.290774 3819 ts_tablet_manager.cc:1428] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:59:22.290980 3534 heartbeater.cc:503] Master 127.17.67.254:33909 was elected leader, sending a full tablet report...
I20260620 21:59:22.291086 3668 heartbeater.cc:503] Master 127.17.67.254:33909 was elected leader, sending a full tablet report...
I20260620 21:59:22.292400 3821 raft_consensus.cc:348] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.292642 3821 raft_consensus.cc:374] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:22.292692 3821 raft_consensus.cc:729] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337b2bdee6e344ea9f8c630744ae1f5f, State: Initialized, Role: FOLLOWER
I20260620 21:59:22.293111 3821 consensus_queue.cc:260] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.293758 3802 heartbeater.cc:503] Master 127.17.67.254:33909 was elected leader, sending a full tablet report...
I20260620 21:59:22.294111 3821 ts_tablet_manager.cc:1428] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:59:22.318835 3825 raft_consensus.cc:482] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:22.319065 3825 raft_consensus.cc:504] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.320184 3825 leader_election.cc:288] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc945746fb14452391fe31f341722244 (127.17.67.194:44107), 337b2bdee6e344ea9f8c630744ae1f5f (127.17.67.195:40581)
I20260620 21:59:22.324544 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.320538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:48817 (local address 127.17.67.195:40581)
0620 21:59:22.320654 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.320659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.320889 (+ 230us) server_negotiation.cc:407] Connection header received
0620 21:59:22.321054 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.321057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.321131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.321210 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:22.321506 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.322241 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.323421 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.323954 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.324052 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.324120 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.324337 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.324418 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:59:22.324558 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.320422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:40581 (local address 127.17.67.193:48817)
0620 21:59:22.320782 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.320795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.320932 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.321231 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.321236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.321255 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:22.321388 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.321395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.322345 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.322349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.323304 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.323313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.324052 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.324056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.324158 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.324226 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.324295 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.324350 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":48,"thread_start_us":114,"threads_started":1}
I20260620 21:59:22.324621 3815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.320498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:33661 (local address 127.17.67.194:44107)
0620 21:59:22.320625 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.320630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.320953 (+ 323us) server_negotiation.cc:407] Connection header received
0620 21:59:22.321115 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.321118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.321170 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.321261 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:22.321688 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.322345 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.323533 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.324036 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.324152 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.324212 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.324426 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.324498 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:22.324589 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.320422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:44107 (local address 127.17.67.193:33661)
0620 21:59:22.320858 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.320870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.320990 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.321360 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.321365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.321385 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:22.321544 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.321551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.322461 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.322464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.323410 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.323420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.324155 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.324159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.324271 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.324318 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.324395 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.324455 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":62,"thread_start_us":115,"threads_started":1}
I20260620 21:59:22.325501 3756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6577d67ba72c40e48c8d2351a035377e" candidate_uuid: "91b085dd32de420698281cdaf27b136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" is_pre_election: true
I20260620 21:59:22.325716 3622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6577d67ba72c40e48c8d2351a035377e" candidate_uuid: "91b085dd32de420698281cdaf27b136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc945746fb14452391fe31f341722244" is_pre_election: true
I20260620 21:59:22.325817 3756 raft_consensus.cc:2454] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b085dd32de420698281cdaf27b136b in term 0.
I20260620 21:59:22.326010 3622 raft_consensus.cc:2454] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b085dd32de420698281cdaf27b136b in term 0.
I20260620 21:59:22.326320 3422 leader_election.cc:302] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337b2bdee6e344ea9f8c630744ae1f5f, 91b085dd32de420698281cdaf27b136b; no voters:
I20260620 21:59:22.326609 3825 raft_consensus.cc:2781] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:22.326694 3825 raft_consensus.cc:482] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:22.326735 3825 raft_consensus.cc:3037] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:22.327751 3825 raft_consensus.cc:504] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.328075 3825 leader_election.cc:288] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [CANDIDATE]: Term 1 election: Requested vote from peers dc945746fb14452391fe31f341722244 (127.17.67.194:44107), 337b2bdee6e344ea9f8c630744ae1f5f (127.17.67.195:40581)
I20260620 21:59:22.328347 3622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6577d67ba72c40e48c8d2351a035377e" candidate_uuid: "91b085dd32de420698281cdaf27b136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc945746fb14452391fe31f341722244"
I20260620 21:59:22.328390 3756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6577d67ba72c40e48c8d2351a035377e" candidate_uuid: "91b085dd32de420698281cdaf27b136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337b2bdee6e344ea9f8c630744ae1f5f"
I20260620 21:59:22.328475 3622 raft_consensus.cc:3037] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:22.328529 3756 raft_consensus.cc:3037] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:22.329497 3622 raft_consensus.cc:2454] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b085dd32de420698281cdaf27b136b in term 1.
I20260620 21:59:22.329506 3756 raft_consensus.cc:2454] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b085dd32de420698281cdaf27b136b in term 1.
I20260620 21:59:22.329828 3422 leader_election.cc:302] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337b2bdee6e344ea9f8c630744ae1f5f, 91b085dd32de420698281cdaf27b136b; no voters:
I20260620 21:59:22.330008 3825 raft_consensus.cc:2781] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:22.330242 3825 raft_consensus.cc:686] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 1 LEADER]: Becoming Leader. State: Replica: 91b085dd32de420698281cdaf27b136b, State: Running, Role: LEADER
I20260620 21:59:22.330541 3825 consensus_queue.cc:237] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } }
I20260620 21:59:22.333833 3341 catalog_manager.cc:5495] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b reported cstate change: term changed from 0 to 1, leader changed from <none> to 91b085dd32de420698281cdaf27b136b (127.17.67.193). New cstate: current_term: 1 leader_uuid: "91b085dd32de420698281cdaf27b136b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b085dd32de420698281cdaf27b136b" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40929 } health_report { overall_health: HEALTHY } } }
I20260620 21:59:22.345355 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.342170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.17.67.193:40929)
0620 21:59:22.342277 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.342281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.342294 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:22.342433 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.342436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.342491 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.342571 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:22.342772 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.343280 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.344096 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.344304 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.344385 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.344655 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:22.345040 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:22.345151 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.345222 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.345254 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:59:22.350988 3756 raft_consensus.cc:1261] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 1 FOLLOWER]: Refusing update from remote peer 91b085dd32de420698281cdaf27b136b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:22.350979 3622 raft_consensus.cc:1261] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 1 FOLLOWER]: Refusing update from remote peer 91b085dd32de420698281cdaf27b136b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:22.351629 3825 consensus_queue.cc:1035] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [LEADER]: Connected to new peer: Peer: permanent_uuid: "337b2bdee6e344ea9f8c630744ae1f5f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:22.351837 3830 consensus_queue.cc:1035] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc945746fb14452391fe31f341722244" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:22.361383 3834 mvcc.cc:205] Tried to move back new op lower bound from 7299042354582446080 to 7299042354508443648. Current Snapshot: MvccSnapshot[applied={T|T < 7299042354582446080 or (T in {7299042354582446080})}]
I20260620 21:59:22.368131 3341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:22.368336 3341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:22.370987 3341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4b07a64dd344735b3edbb062fb53100: Sending DeleteTablet for 3 replicas of tablet 6577d67ba72c40e48c8d2351a035377e
I20260620 21:59:22.371809 3736 tablet_service.cc:1514] Processing DeleteTablet for tablet 6577d67ba72c40e48c8d2351a035377e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:22 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:56206
I20260620 21:59:22.371929 3602 tablet_service.cc:1514] Processing DeleteTablet for tablet 6577d67ba72c40e48c8d2351a035377e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:22 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:43338
I20260620 21:59:22.371975 3468 tablet_service.cc:1514] Processing DeleteTablet for tablet 6577d67ba72c40e48c8d2351a035377e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:22 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:45314
I20260620 21:59:22.372452 3843 tablet_replica.cc:330] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: stopping tablet replica
I20260620 21:59:22.372455 3844 tablet_replica.cc:330] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b: stopping tablet replica
I20260620 21:59:22.372766 3843 raft_consensus.cc:2229] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:22.372762 3844 raft_consensus.cc:2229] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:22.373075 3843 raft_consensus.cc:2258] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:22.373108 3844 raft_consensus.cc:2258] T 6577d67ba72c40e48c8d2351a035377e P 91b085dd32de420698281cdaf27b136b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:22.373215 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3405
I20260620 21:59:22.373956 3842 tablet_replica.cc:330] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: stopping tablet replica
I20260620 21:59:22.374264 3842 raft_consensus.cc:2229] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:22.374557 3842 raft_consensus.cc:2258] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:22.375468 3843 ts_tablet_manager.cc:1905] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:22.375928 3842 ts_tablet_manager.cc:1905] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:22.378016 3843 ts_tablet_manager.cc:1918] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:22.378140 3843 log.cc:1196] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/6577d67ba72c40e48c8d2351a035377e
I20260620 21:59:22.378458 3843 ts_tablet_manager.cc:1939] T 6577d67ba72c40e48c8d2351a035377e P dc945746fb14452391fe31f341722244: Deleting consensus metadata
I20260620 21:59:22.379264 3842 ts_tablet_manager.cc:1918] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:22.379364 3842 log.cc:1196] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/6577d67ba72c40e48c8d2351a035377e
I20260620 21:59:22.379596 3330 catalog_manager.cc:4841] TS dc945746fb14452391fe31f341722244 (127.17.67.194:44107): tablet 6577d67ba72c40e48c8d2351a035377e (table test-table [id=fdb07775c24d408187b3dd2019983642]) successfully deleted
I20260620 21:59:22.379846 3842 ts_tablet_manager.cc:1939] T 6577d67ba72c40e48c8d2351a035377e P 337b2bdee6e344ea9f8c630744ae1f5f: Deleting consensus metadata
I20260620 21:59:22.381192 3328 catalog_manager.cc:4841] TS 337b2bdee6e344ea9f8c630744ae1f5f (127.17.67.195:40581): tablet 6577d67ba72c40e48c8d2351a035377e (table test-table [id=fdb07775c24d408187b3dd2019983642]) successfully deleted
W20260620 21:59:22.382051 3330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:40929 (error 108)
I20260620 21:59:22.382817 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3539
I20260620 21:59:22.383188 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.382815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40929 (local address 127.0.0.1:45330)
0620 21:59:22.383001 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.383077 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 21:59:22.384429 3330 catalog_manager.cc:4584] TS 91b085dd32de420698281cdaf27b136b (127.17.67.193:40929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6577d67ba72c40e48c8d2351a035377e: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40929: connect: Connection refused (error 111)
I20260620 21:59:22.388976 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3673
I20260620 21:59:22.394845 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3311
2026-06-20T21:59:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2303 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T21:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:22Z Disabled control of system clock
I20260620 21:59:22.424872 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:40901
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:22.531474 3849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:22.531723 3849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:22.531757 3849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:22.535605 3849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:22.535656 3849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:22.535671 3849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:22.535687 3849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:22.535703 3849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:22.540489 3849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:40901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:22.541535 3849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:22.542913 3849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:22.551385 3854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:22.551422 3857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:22.551421 3855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:22.551631 3849 server_base.cc:1034] running on GCE node
I20260620 21:59:22.552402 3849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:22.553496 3849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:22.554708 3849 hybrid_clock.cc:648] HybridClock initialized: now 1781992762554680 us; error 41 us; skew 500 ppm
I20260620 21:59:22.556581 3849 webserver.cc:460] Webserver started at http://127.17.67.254:45781/ using document root <none> and password file <none>
I20260620 21:59:22.557164 3849 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:22.557260 3849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:22.557483 3849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:22.559549 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "fb0e8327e54b4b5aa0cc3647adbef93d"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "e65062424977cd2d748986cd687cd878"
server_key_iv: "09a81a9f38c97290981bb731af18b49d"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.560135 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "fb0e8327e54b4b5aa0cc3647adbef93d"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "e65062424977cd2d748986cd687cd878"
server_key_iv: "09a81a9f38c97290981bb731af18b49d"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.563684 3849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:22.566035 3863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.567221 3849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:22.567358 3849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "fb0e8327e54b4b5aa0cc3647adbef93d"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "e65062424977cd2d748986cd687cd878"
server_key_iv: "09a81a9f38c97290981bb731af18b49d"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.567468 3849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:22.621125 3849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:22.621930 3849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:22.622146 3849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:22.629240 3849 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:40901
I20260620 21:59:22.629266 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:40901 every 8 connection(s)
I20260620 21:59:22.630293 3849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:22.630937 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3849
I20260620 21:59:22.631072 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:22.631352 17679 external_mini_cluster.cc:1428] Setting key cc7a4868635de7075ea3ace74256f252
I20260620 21:59:22.633838 3916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:22.639343 3916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: Bootstrap starting.
I20260620 21:59:22.640740 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.632779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.17.67.254:40901)
0620 21:59:22.633507 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.633542 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.633606 (+ 64us) server_negotiation.cc:407] Connection header received
0620 21:59:22.634333 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.634352 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.634745 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.635095 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.635522 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.636486 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.637376 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.637906 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.638081 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.638981 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:22.638986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:22.638998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:22.639415 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:22.639510 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:22.639540 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:22.639654 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.639795 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.640043 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
I20260620 21:59:22.641633 3916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:22.642513 3916 log.cc:824] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:22.644447 3916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: No bootstrap required, opened a new log
I20260620 21:59:22.647866 3916 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } }
I20260620 21:59:22.648126 3916 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:22.648166 3916 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0e8327e54b4b5aa0cc3647adbef93d, State: Initialized, Role: FOLLOWER
I20260620 21:59:22.648772 3916 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } }
I20260620 21:59:22.648941 3916 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:22.648993 3916 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:22.649113 3916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:22.650197 3916 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } }
I20260620 21:59:22.650487 3916 leader_election.cc:302] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0e8327e54b4b5aa0cc3647adbef93d; no voters:
I20260620 21:59:22.650722 3916 leader_election.cc:288] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:22.650846 3921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:22.651093 3921 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [term 1 LEADER]: Becoming Leader. State: Replica: fb0e8327e54b4b5aa0cc3647adbef93d, State: Running, Role: LEADER
I20260620 21:59:22.651381 3921 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } }
I20260620 21:59:22.651952 3916 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:22.653012 3922 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } } }
I20260620 21:59:22.653151 3922 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:22.654213 3923 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0e8327e54b4b5aa0cc3647adbef93d. Latest consensus state: current_term: 1 leader_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e8327e54b4b5aa0cc3647adbef93d" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 40901 } } }
I20260620 21:59:22.654659 3923 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:22.656864 3937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:22.656929 3937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:22.657011 3930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:22.657763 3930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:22.662931 3930 catalog_manager.cc:1347] Generated new cluster ID: 4e507af95d3b4458a94b98ce79d6999c
I20260620 21:59:22.663005 3930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:22.671144 3930 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:22.672374 3930 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:22.678224 3930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: Generated new TSK 0
I20260620 21:59:22.679006 3930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:22.690815 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--builtin_ntp_servers=127.17.67.212:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:22.807536 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:22.807787 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:22.807827 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:22.811815 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:22.811865 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:22.811919 3940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:22.816496 3940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:22.817514 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:22.818881 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:22.828182 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:22.828327 3940 server_base.cc:1034] running on GCE node
W20260620 21:59:22.828169 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:22.828197 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:22.828858 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:22.829488 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:22.830722 3940 hybrid_clock.cc:648] HybridClock initialized: now 1781992762830709 us; error 24 us; skew 500 ppm
I20260620 21:59:22.832583 3940 webserver.cc:460] Webserver started at http://127.17.67.193:46253/ using document root <none> and password file <none>
I20260620 21:59:22.833158 3940 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:22.833251 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:22.833472 3940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:22.835459 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "6e29bc7bad6d4886a92f0d297f93b4ae"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "c0cf197d911542f43041d08c6d843f2e"
server_key_iv: "13e75ec201e56f57335745311156e1f0"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.836030 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "6e29bc7bad6d4886a92f0d297f93b4ae"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "c0cf197d911542f43041d08c6d843f2e"
server_key_iv: "13e75ec201e56f57335745311156e1f0"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.839592 3940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:22.842020 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.843230 3940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:22.843372 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "6e29bc7bad6d4886a92f0d297f93b4ae"
format_stamp: "Formatted at 2026-06-20 21:59:22 on dist-test-slave-cx8d"
server_key: "c0cf197d911542f43041d08c6d843f2e"
server_key_iv: "13e75ec201e56f57335745311156e1f0"
server_key_version: "encryptionkey@0"
I20260620 21:59:22.843487 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:22.852674 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:22.853390 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:22.853643 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:22.854200 3940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:22.855096 3940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:22.855168 3940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.855225 3940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:22.855275 3940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:22.863804 3940 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:36011
I20260620 21:59:22.863835 4067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:36011 every 8 connection(s)
I20260620 21:59:22.864647 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:22.867516 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 3940
I20260620 21:59:22.867653 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:22.867952 17679 external_mini_cluster.cc:1428] Setting key eae53357bb3f68de1a6bfaa647ae1504
I20260620 21:59:22.870224 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--builtin_ntp_servers=127.17.67.212:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:22.875738 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.866486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:54271 (local address 127.17.67.254:40901)
0620 21:59:22.866624 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:22.866628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:22.867764 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 21:59:22.869519 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:22.869522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:22.869616 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:22.869711 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:22.870720 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.871581 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.873356 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.873784 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.873906 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.875106 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:22.875113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:22.875119 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:22.875138 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:22.875164 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:22.875177 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:22.875264 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:22.875414 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 21:59:22.875525 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:59:22.876463 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:22.866876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40901 (local address 127.17.67.193:54271)
0620 21:59:22.867610 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:22.867665 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:22.869243 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:22.869864 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:22.869871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:22.869951 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:22.870516 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.870571 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.871797 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.871804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.873227 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:22.873236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:22.873953 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:22.873957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:22.874149 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:22.874820 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:22.874822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:22.874861 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:22.874923 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:22.875276 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:22.875283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:22.875287 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:22.875376 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:59:22.875619 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":130,"threads_started":1}
I20260620 21:59:22.877725 4068 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40901
I20260620 21:59:22.877962 4068 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:22.878466 4068 heartbeater.cc:511] Master 127.17.67.254:40901 requested a full tablet report, sending...
I20260620 21:59:22.879909 3879 ts_manager.cc:194] Registered new tserver with Master: 6e29bc7bad6d4886a92f0d297f93b4ae (127.17.67.193:36011)
I20260620 21:59:22.881031 3879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.193:54271
W20260620 21:59:22.982997 4071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:22.983263 4071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:22.983304 4071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:22.987303 4071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:22.987355 4071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:22.987468 4071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:22.992281 4071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:22.993381 4071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:22.994819 4071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:23.004827 4077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:23.004827 4076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:23.004833 4079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:23.004869 4071 server_base.cc:1034] running on GCE node
I20260620 21:59:23.005649 4071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:23.006320 4071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:23.007603 4071 hybrid_clock.cc:648] HybridClock initialized: now 1781992763007583 us; error 32 us; skew 500 ppm
I20260620 21:59:23.009552 4071 webserver.cc:460] Webserver started at http://127.17.67.194:44295/ using document root <none> and password file <none>
I20260620 21:59:23.010226 4071 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:23.010329 4071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:23.010601 4071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:23.012786 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "8c7a47993fde4404984f17741cd267ab"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "f7166850e72852e161cdad2c5be86506"
server_key_iv: "cda418f511a4bf39e07e0982d3643649"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.013414 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "8c7a47993fde4404984f17741cd267ab"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "f7166850e72852e161cdad2c5be86506"
server_key_iv: "cda418f511a4bf39e07e0982d3643649"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.017398 4071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 21:59:23.020028 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.021327 4071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:23.021469 4071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "8c7a47993fde4404984f17741cd267ab"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "f7166850e72852e161cdad2c5be86506"
server_key_iv: "cda418f511a4bf39e07e0982d3643649"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.021625 4071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:23.042268 4071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:23.043018 4071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:23.043248 4071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:23.043882 4071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:23.044895 4071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:23.044972 4071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.045055 4071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:23.045104 4071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.055047 4071 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:43807
I20260620 21:59:23.055065 4198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:43807 every 8 connection(s)
I20260620 21:59:23.056000 4071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:23.057622 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4071
I20260620 21:59:23.057760 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:23.058120 17679 external_mini_cluster.cc:1428] Setting key dd3c427acd0278cb4be7870671c24f2c
I20260620 21:59:23.060163 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--builtin_ntp_servers=127.17.67.212:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:23.065882 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.057908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:58431 (local address 127.17.67.254:40901)
0620 21:59:23.058076 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.058081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.059382 (+ 1301us) server_negotiation.cc:407] Connection header received
0620 21:59:23.060853 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.060856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.060911 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.060991 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.062047 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.062578 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.063735 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.063956 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.064069 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.065209 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:23.065213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:23.065216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:23.065228 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:23.065252 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:23.065261 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:23.065341 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.065586 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.065665 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:59:23.066636 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.058412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40901 (local address 127.17.67.194:58431)
0620 21:59:23.059234 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.059287 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.060637 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.061164 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.061172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.061261 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:23.061891 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.061913 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.062721 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.062724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.063603 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.063612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.064107 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.064111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.064296 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.064930 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:23.064932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:23.064972 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:23.065043 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:23.065343 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:23.065355 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:23.065364 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.065526 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.065795 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":142,"threads_started":1}
I20260620 21:59:23.067833 4199 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40901
I20260620 21:59:23.068089 4199 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:23.068595 4199 heartbeater.cc:511] Master 127.17.67.254:40901 requested a full tablet report, sending...
I20260620 21:59:23.069672 3879 ts_manager.cc:194] Registered new tserver with Master: 8c7a47993fde4404984f17741cd267ab (127.17.67.194:43807)
I20260620 21:59:23.070276 3879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.194:58431
W20260620 21:59:23.172168 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:23.172482 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:23.172519 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:23.176589 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:23.176640 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:23.176694 4202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:23.181610 4202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:40901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:23.182776 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:23.184163 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:23.193857 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:23.193861 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:23.193861 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:23.194200 4202 server_base.cc:1034] running on GCE node
I20260620 21:59:23.194693 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:23.195410 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:23.196636 4202 hybrid_clock.cc:648] HybridClock initialized: now 1781992763196621 us; error 31 us; skew 500 ppm
I20260620 21:59:23.198534 4202 webserver.cc:460] Webserver started at http://127.17.67.195:32901/ using document root <none> and password file <none>
I20260620 21:59:23.199180 4202 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:23.199275 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:23.199506 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:23.201539 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "e01f9f9b809b48a2bc3dab2bc866a03c"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "d4e8273589eb4bacca17c451932c0ed3"
server_key_iv: "fc1780f060703899ecc97e950850c03a"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.202168 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "e01f9f9b809b48a2bc3dab2bc866a03c"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "d4e8273589eb4bacca17c451932c0ed3"
server_key_iv: "fc1780f060703899ecc97e950850c03a"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.205865 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:23.208323 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.209640 4202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:23.209769 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "e01f9f9b809b48a2bc3dab2bc866a03c"
format_stamp: "Formatted at 2026-06-20 21:59:23 on dist-test-slave-cx8d"
server_key: "d4e8273589eb4bacca17c451932c0ed3"
server_key_iv: "fc1780f060703899ecc97e950850c03a"
server_key_version: "encryptionkey@0"
I20260620 21:59:23.209882 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:23.222544 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:23.223222 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:23.223485 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:23.224110 4202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:23.225217 4202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:23.225297 4202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.225385 4202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:23.225425 4202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:23.235105 4202 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:37427
I20260620 21:59:23.235131 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:37427 every 8 connection(s)
I20260620 21:59:23.236064 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:23.236310 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4202
I20260620 21:59:23.236405 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:23.236760 17679 external_mini_cluster.cc:1428] Setting key fec20d1fa3c16186e03dee7bb90624f9
I20260620 21:59:23.245615 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.238101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:46013 (local address 127.17.67.254:40901)
0620 21:59:23.238258 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.238262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.239624 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 21:59:23.240954 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.240957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.241005 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.241081 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.242099 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.242627 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.243754 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.244013 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.244107 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.245155 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:23.245159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:23.245161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:23.245173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:23.245191 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:23.245199 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:23.245261 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.245403 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.245453 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:59:23.246472 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.238608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:40901 (local address 127.17.67.195:46013)
0620 21:59:23.239423 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.239488 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.240753 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.241241 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.241247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.241330 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:23.241941 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.241957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.242738 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.242740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.243617 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.243625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.244154 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.244156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.244291 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.244891 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:23.244893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:23.244932 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:23.244992 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:23.245274 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:23.245280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:23.245284 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.245375 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.245630 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":145,"threads_started":1}
I20260620 21:59:23.247699 4330 heartbeater.cc:347] Connected to a master server at 127.17.67.254:40901
I20260620 21:59:23.247957 4330 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:23.248489 4330 heartbeater.cc:511] Master 127.17.67.254:40901 requested a full tablet report, sending...
I20260620 21:59:23.249529 3879 ts_manager.cc:194] Registered new tserver with Master: e01f9f9b809b48a2bc3dab2bc866a03c (127.17.67.195:37427)
I20260620 21:59:23.250223 3879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.195:46013
I20260620 21:59:23.250957 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:23.256696 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.253021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.17.67.254:40901)
0620 21:59:23.253154 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.253157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.253372 (+ 215us) server_negotiation.cc:407] Connection header received
0620 21:59:23.253478 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.253481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.253518 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.253647 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:23.253865 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.254383 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.255182 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.255413 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.255503 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.256281 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:23.256284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:23.256286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:23.256298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:23.256315 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:23.256322 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:23.256431 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.256562 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.256591 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:59:23.259801 3879 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:23.262787 3879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:23.277382 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.17.67.193:36011)
0620 21:59:23.272629 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.272643 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.272671 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:59:23.272757 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.272762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.272955 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.273095 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:23.273298 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.274282 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.276156 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.276699 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.276827 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.277027 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.277124 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.277220 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":119,"threads_started":1}
I20260620 21:59:23.277379 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:36011 (local address 127.0.0.1:44694)
0620 21:59:23.272073 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.272087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.272181 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.273087 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.273089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.273104 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:23.273177 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.273181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.274382 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.274388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.276015 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.276025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.276826 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.276829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.276932 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.276978 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.277096 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.277214 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":144,"thread_start_us":99,"threads_started":1}
I20260620 21:59:23.277940 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.17.67.194:43807)
0620 21:59:23.272220 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.272234 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.272260 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:59:23.272335 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.272340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.272562 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.273085 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:23.273227 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.274357 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.276145 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.276924 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.277447 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.277650 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.277704 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.277792 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":116,"threads_started":1}
I20260620 21:59:23.277642 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:37427 (local address 127.0.0.1:60936)
0620 21:59:23.272362 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.272380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.272480 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.273847 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.273850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.273869 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:23.273956 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.273961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.275073 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.275076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.276036 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.276046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.277273 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.277276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.277346 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.277365 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.277427 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.277483 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260620 21:59:23.277974 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.17.67.195:37427)
0620 21:59:23.273374 (+ 1623us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.273384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.273406 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:23.273476 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.273481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.273710 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.273854 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:23.274073 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.274939 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.276193 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.276927 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.277042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.277217 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.277732 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.277815 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1524,"thread_start_us":136,"threads_started":1}
I20260620 21:59:23.277676 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.271112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43807 (local address 127.0.0.1:51386)
0620 21:59:23.271961 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.272012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.272143 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.272744 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.272750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.272773 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:23.272912 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.272919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.274446 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.274449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.276014 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.276025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.277078 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.277082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.277165 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.277187 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.277240 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.277311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":116,"thread_start_us":82,"threads_started":1}
I20260620 21:59:23.280712 4133 tablet_service.cc:1467] Processing CreateTablet for tablet c9f6313d5270435e9bb8df851dfa8490 (DEFAULT_TABLE table=test-table [id=8d323d833b4247c6a3beb40e1c6a8782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:23.280753 4264 tablet_service.cc:1467] Processing CreateTablet for tablet c9f6313d5270435e9bb8df851dfa8490 (DEFAULT_TABLE table=test-table [id=8d323d833b4247c6a3beb40e1c6a8782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:23.280754 4002 tablet_service.cc:1467] Processing CreateTablet for tablet c9f6313d5270435e9bb8df851dfa8490 (DEFAULT_TABLE table=test-table [id=8d323d833b4247c6a3beb40e1c6a8782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:23.281746 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f6313d5270435e9bb8df851dfa8490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:23.281752 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f6313d5270435e9bb8df851dfa8490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:23.281752 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f6313d5270435e9bb8df851dfa8490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:23.288053 4347 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Bootstrap starting.
I20260620 21:59:23.289083 4348 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Bootstrap starting.
I20260620 21:59:23.289856 4349 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Bootstrap starting.
I20260620 21:59:23.290693 4347 tablet_bootstrap.cc:654] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:23.291152 4348 tablet_bootstrap.cc:654] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:23.291560 4349 tablet_bootstrap.cc:654] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:23.291566 4347 log.cc:824] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:23.291929 4348 log.cc:824] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:23.292302 4349 log.cc:824] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:23.293628 4347 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: No bootstrap required, opened a new log
I20260620 21:59:23.293808 4347 ts_tablet_manager.cc:1397] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:23.294068 4348 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: No bootstrap required, opened a new log
I20260620 21:59:23.294152 4349 tablet_bootstrap.cc:492] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: No bootstrap required, opened a new log
I20260620 21:59:23.294262 4348 ts_tablet_manager.cc:1397] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:59:23.294342 4349 ts_tablet_manager.cc:1397] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:59:23.297250 4348 raft_consensus.cc:348] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.297523 4348 raft_consensus.cc:374] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:23.297597 4348 raft_consensus.cc:729] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01f9f9b809b48a2bc3dab2bc866a03c, State: Initialized, Role: FOLLOWER
I20260620 21:59:23.298069 4348 consensus_queue.cc:260] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.298175 4347 raft_consensus.cc:348] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.298229 4349 raft_consensus.cc:348] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.298432 4347 raft_consensus.cc:374] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:23.298431 4349 raft_consensus.cc:374] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:23.298492 4347 raft_consensus.cc:729] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7a47993fde4404984f17741cd267ab, State: Initialized, Role: FOLLOWER
I20260620 21:59:23.298501 4349 raft_consensus.cc:729] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e29bc7bad6d4886a92f0d297f93b4ae, State: Initialized, Role: FOLLOWER
I20260620 21:59:23.298846 4330 heartbeater.cc:503] Master 127.17.67.254:40901 was elected leader, sending a full tablet report...
I20260620 21:59:23.298900 4347 consensus_queue.cc:260] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.298929 4349 consensus_queue.cc:260] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.299078 4348 ts_tablet_manager.cc:1428] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 21:59:23.299825 4347 ts_tablet_manager.cc:1428] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:59:23.299825 4349 ts_tablet_manager.cc:1428] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 21:59:23.300140 4199 heartbeater.cc:503] Master 127.17.67.254:40901 was elected leader, sending a full tablet report...
I20260620 21:59:23.300140 4068 heartbeater.cc:503] Master 127.17.67.254:40901 was elected leader, sending a full tablet report...
W20260620 21:59:23.306928 4200 tablet.cc:2369] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:59:23.365885 4069 tablet.cc:2369] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:23.400660 4353 raft_consensus.cc:482] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:23.400919 4353 raft_consensus.cc:504] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.402143 4353 leader_election.cc:288] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c7a47993fde4404984f17741cd267ab (127.17.67.194:43807), 6e29bc7bad6d4886a92f0d297f93b4ae (127.17.67.193:36011)
I20260620 21:59:23.405669 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.402430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:41625 (local address 127.17.67.194:43807)
0620 21:59:23.402558 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.402562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.402577 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:59:23.402731 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.402734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.402800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.402899 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:23.403183 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.403687 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.404705 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.405105 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.405239 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.405288 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.405460 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.405534 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:59:23.405669 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.402364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:43807 (local address 127.17.67.195:41625)
0620 21:59:23.402484 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.402499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.402593 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.402912 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.402915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.402931 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:23.403069 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.403075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.403807 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.403811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.404580 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.404588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.405231 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.405235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.405332 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.405357 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.405446 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.405514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"spinlock_wait_cycles":2944}
I20260620 21:59:23.405855 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.402555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:59969 (local address 127.17.67.193:36011)
0620 21:59:23.402696 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.402702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.402928 (+ 226us) server_negotiation.cc:407] Connection header received
0620 21:59:23.403093 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.403100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.403141 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.403228 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:23.403488 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.403965 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.404971 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.405338 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.405464 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.405508 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.405683 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.405746 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:59:23.405874 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.402372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:36011 (local address 127.17.67.195:59969)
0620 21:59:23.402832 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.402842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:23.402954 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:23.403237 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:23.403239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:23.403254 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:23.403375 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.403380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.404096 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.404099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.404843 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:23.404852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.405455 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.405458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:23.405555 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.405597 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:23.405670 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:59:23.405733 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":135,"thread_start_us":135,"threads_started":1}
I20260620 21:59:23.406466 4022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f6313d5270435e9bb8df851dfa8490" candidate_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" is_pre_election: true
I20260620 21:59:23.406474 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f6313d5270435e9bb8df851dfa8490" candidate_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7a47993fde4404984f17741cd267ab" is_pre_election: true
I20260620 21:59:23.406750 4153 raft_consensus.cc:2454] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f9f9b809b48a2bc3dab2bc866a03c in term 0.
I20260620 21:59:23.406759 4022 raft_consensus.cc:2454] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01f9f9b809b48a2bc3dab2bc866a03c in term 0.
I20260620 21:59:23.407223 4218 leader_election.cc:302] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e29bc7bad6d4886a92f0d297f93b4ae, e01f9f9b809b48a2bc3dab2bc866a03c; no voters:
I20260620 21:59:23.407461 4353 raft_consensus.cc:2781] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:23.407548 4353 raft_consensus.cc:482] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:23.407591 4353 raft_consensus.cc:3037] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:23.408592 4353 raft_consensus.cc:504] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.408900 4353 leader_election.cc:288] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [CANDIDATE]: Term 1 election: Requested vote from peers 8c7a47993fde4404984f17741cd267ab (127.17.67.194:43807), 6e29bc7bad6d4886a92f0d297f93b4ae (127.17.67.193:36011)
I20260620 21:59:23.409235 4022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f6313d5270435e9bb8df851dfa8490" candidate_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae"
I20260620 21:59:23.409235 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f6313d5270435e9bb8df851dfa8490" candidate_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7a47993fde4404984f17741cd267ab"
I20260620 21:59:23.409368 4153 raft_consensus.cc:3037] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:23.409343 4022 raft_consensus.cc:3037] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:23.410398 4022 raft_consensus.cc:2454] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f9f9b809b48a2bc3dab2bc866a03c in term 1.
I20260620 21:59:23.410580 4153 raft_consensus.cc:2454] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01f9f9b809b48a2bc3dab2bc866a03c in term 1.
I20260620 21:59:23.410745 4218 leader_election.cc:302] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e29bc7bad6d4886a92f0d297f93b4ae, e01f9f9b809b48a2bc3dab2bc866a03c; no voters:
I20260620 21:59:23.410907 4353 raft_consensus.cc:2781] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:23.411118 4353 raft_consensus.cc:686] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 1 LEADER]: Becoming Leader. State: Replica: e01f9f9b809b48a2bc3dab2bc866a03c, State: Running, Role: LEADER
I20260620 21:59:23.411458 4353 consensus_queue.cc:237] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } }
I20260620 21:59:23.414837 3879 catalog_manager.cc:5495] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c reported cstate change: term changed from 0 to 1, leader changed from <none> to e01f9f9b809b48a2bc3dab2bc866a03c (127.17.67.195). New cstate: current_term: 1 leader_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01f9f9b809b48a2bc3dab2bc866a03c" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 37427 } health_report { overall_health: HEALTHY } } }
I20260620 21:59:23.429108 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.425380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.17.67.195:37427)
0620 21:59:23.425498 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:23.425501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:23.425514 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:23.425708 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:23.425710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:23.425766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:23.425858 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:23.426254 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.426842 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.427785 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:23.427985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:23.428067 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:23.428343 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:23.428754 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:23.428862 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:23.428971 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 21:59:23.429001 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:59:23.434799 4153 raft_consensus.cc:1261] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 1 FOLLOWER]: Refusing update from remote peer e01f9f9b809b48a2bc3dab2bc866a03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:23.434841 4022 raft_consensus.cc:1261] T c9f6313d5270435e9bb8df851dfa8490 P 6e29bc7bad6d4886a92f0d297f93b4ae [term 1 FOLLOWER]: Refusing update from remote peer e01f9f9b809b48a2bc3dab2bc866a03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:23.435467 4353 consensus_queue.cc:1035] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e29bc7bad6d4886a92f0d297f93b4ae" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:23.435642 4357 consensus_queue.cc:1035] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7a47993fde4404984f17741cd267ab" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:23.442863 4360 mvcc.cc:205] Tried to move back new op lower bound from 7299042359021977600 to 7299042358936289280. Current Snapshot: MvccSnapshot[applied={T|T < 7299042359021977600}]
I20260620 21:59:23.443085 4361 mvcc.cc:205] Tried to move back new op lower bound from 7299042359021977600 to 7299042358936289280. Current Snapshot: MvccSnapshot[applied={T|T < 7299042359021977600}]
I20260620 21:59:23.444314 4362 mvcc.cc:205] Tried to move back new op lower bound from 7299042359021977600 to 7299042358936289280. Current Snapshot: MvccSnapshot[applied={T|T < 7299042359021977600}]
I20260620 21:59:23.452018 3877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:23.452162 3877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:23.454860 3877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb0e8327e54b4b5aa0cc3647adbef93d: Sending DeleteTablet for 3 replicas of tablet c9f6313d5270435e9bb8df851dfa8490
I20260620 21:59:23.455634 4133 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f6313d5270435e9bb8df851dfa8490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:23 UTC) from {username='slave'} at 127.0.0.1:51386
I20260620 21:59:23.455634 4002 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f6313d5270435e9bb8df851dfa8490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:23 UTC) from {username='slave'} at 127.0.0.1:44694
I20260620 21:59:23.455654 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f6313d5270435e9bb8df851dfa8490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:23 UTC) from {username='slave'} at 127.0.0.1:60936
I20260620 21:59:23.456187 4370 tablet_replica.cc:330] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: stopping tablet replica
I20260620 21:59:23.456436 4372 tablet_replica.cc:330] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: stopping tablet replica
I20260620 21:59:23.456802 4370 raft_consensus.cc:2229] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:23.457005 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3940
I20260620 21:59:23.458212 4372 raft_consensus.cc:2229] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:23.458722 4370 raft_consensus.cc:2258] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:23.459093 4372 raft_consensus.cc:2258] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:23.460721 4372 ts_tablet_manager.cc:1905] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:23.462466 4370 ts_tablet_manager.cc:1905] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:23.463068 4372 ts_tablet_manager.cc:1918] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:23.463179 4372 log.cc:1196] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/c9f6313d5270435e9bb8df851dfa8490
I20260620 21:59:23.463449 4372 ts_tablet_manager.cc:1939] T c9f6313d5270435e9bb8df851dfa8490 P 8c7a47993fde4404984f17741cd267ab: Deleting consensus metadata
I20260620 21:59:23.464565 3867 catalog_manager.cc:4841] TS 8c7a47993fde4404984f17741cd267ab (127.17.67.194:43807): tablet c9f6313d5270435e9bb8df851dfa8490 (table test-table [id=8d323d833b4247c6a3beb40e1c6a8782]) successfully deleted
I20260620 21:59:23.464906 4370 ts_tablet_manager.cc:1918] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:23.464996 4370 log.cc:1196] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/c9f6313d5270435e9bb8df851dfa8490
I20260620 21:59:23.465286 4370 ts_tablet_manager.cc:1939] T c9f6313d5270435e9bb8df851dfa8490 P e01f9f9b809b48a2bc3dab2bc866a03c: Deleting consensus metadata
I20260620 21:59:23.466355 3866 catalog_manager.cc:4841] TS e01f9f9b809b48a2bc3dab2bc866a03c (127.17.67.195:37427): tablet c9f6313d5270435e9bb8df851dfa8490 (table test-table [id=8d323d833b4247c6a3beb40e1c6a8782]) successfully deleted
W20260620 21:59:23.469720 3865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:36011 (error 108)
I20260620 21:59:23.470011 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4071
I20260620 21:59:23.471814 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:23.471472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:36011 (local address 127.0.0.1:44700)
0620 21:59:23.471627 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:23.471710 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:59:23.472431 3865 catalog_manager.cc:4584] TS 6e29bc7bad6d4886a92f0d297f93b4ae (127.17.67.193:36011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9f6313d5270435e9bb8df851dfa8490: Network error: Client connection negotiation failed: client connection to 127.17.67.193:36011: connect: Connection refused (error 111)
I20260620 21:59:23.476182 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4202
I20260620 21:59:23.483071 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 3849
2026-06-20T21:59:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:23 dist-test-slave-cx8d krb5kdc[4376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:23 dist-test-slave-cx8d krb5kdc[4376](info): set up 2 sockets
Jun 20 21:59:23 dist-test-slave-cx8d krb5kdc[4376](info): commencing operation
krb5kdc: starting...
W20260620 21:59:24.573526 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:24 dist-test-slave-cx8d krb5kdc[4376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:24.692710 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38951
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:38951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:24.803939 4389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:24.804272 4389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:24.804352 4389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:24.808391 4389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:24.808490 4389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:24.808527 4389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:24.808561 4389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:24.808597 4389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:24.813982 4389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:38951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:24.815097 4389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:24.816215 4389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:24.824967 4397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:24.825001 4395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:24.824971 4394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:24.825235 4389 server_base.cc:1034] running on GCE node
I20260620 21:59:24.825947 4389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:24.827060 4389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:24.828233 4389 hybrid_clock.cc:648] HybridClock initialized: now 1781992764828206 us; error 51 us; skew 500 ppm
Jun 20 21:59:24 dist-test-slave-cx8d krb5kdc[4376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992764, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:24.831405 4389 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:24.832304 4389 webserver.cc:460] Webserver started at http://127.17.67.254:43177/ using document root <none> and password file <none>
I20260620 21:59:24.832882 4389 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:24.832978 4389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:24.833212 4389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:24.835317 4389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "220755250ef641deab48e701d4f9c160"
format_stamp: "Formatted at 2026-06-20 21:59:24 on dist-test-slave-cx8d"
server_key: "560f138126b2bf5d63b8ecb3303207fa"
server_key_iv: "e38da66ea72eaa0937dc210ff1c57b09"
server_key_version: "encryptionkey@0"
I20260620 21:59:24.835901 4389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "220755250ef641deab48e701d4f9c160"
format_stamp: "Formatted at 2026-06-20 21:59:24 on dist-test-slave-cx8d"
server_key: "560f138126b2bf5d63b8ecb3303207fa"
server_key_iv: "e38da66ea72eaa0937dc210ff1c57b09"
server_key_version: "encryptionkey@0"
I20260620 21:59:24.839720 4389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:59:24.842336 4404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:24.843742 4389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:24.843938 4389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "220755250ef641deab48e701d4f9c160"
format_stamp: "Formatted at 2026-06-20 21:59:24 on dist-test-slave-cx8d"
server_key: "560f138126b2bf5d63b8ecb3303207fa"
server_key_iv: "e38da66ea72eaa0937dc210ff1c57b09"
server_key_version: "encryptionkey@0"
I20260620 21:59:24.844084 4389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:24.855293 4389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:24.859086 4389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:24.859324 4389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:24.866744 4389 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:38951
I20260620 21:59:24.866752 4456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:38951 every 8 connection(s)
I20260620 21:59:24.867856 4389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:24.868968 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4389
I20260620 21:59:24.869104 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:24.869436 17679 external_mini_cluster.cc:1428] Setting key 7c2539ab0c9895774992c6991a182dd0
I20260620 21:59:24.871317 4457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:24.877998 4457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: Bootstrap starting.
Jun 20 21:59:24 dist-test-slave-cx8d krb5kdc[4376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:24.880685 4457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:24.881639 4457 log.cc:824] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:24.884085 4457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: No bootstrap required, opened a new log
I20260620 21:59:24.885211 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:24.870854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.17.67.254:38951)
0620 21:59:24.871289 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:24.871308 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:24.871352 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:59:24.872183 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:24.872204 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:24.872599 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:24.872952 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:24.873720 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.875205 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.876147 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:24.876833 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:24.877018 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:24.879822 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:24.879828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:24.879843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:24.879876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:24.881937 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:24.882533 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:24.882564 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:24.882571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:24.882710 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:24.883027 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:24.883030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:24.883031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:24.883521 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:24.883705 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:24.883976 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 21:59:24.884281 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260620 21:59:24.888222 4457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } }
I20260620 21:59:24.888458 4457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:24.888509 4457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220755250ef641deab48e701d4f9c160, State: Initialized, Role: FOLLOWER
I20260620 21:59:24.888862 4457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } }
I20260620 21:59:24.888970 4457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:24.889015 4457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:24.889083 4457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:24.890024 4457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } }
I20260620 21:59:24.890401 4457 leader_election.cc:302] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220755250ef641deab48e701d4f9c160; no voters:
I20260620 21:59:24.890774 4457 leader_election.cc:288] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:24.890846 4462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:24.891057 4462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [term 1 LEADER]: Becoming Leader. State: Replica: 220755250ef641deab48e701d4f9c160, State: Running, Role: LEADER
I20260620 21:59:24.891347 4462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } }
I20260620 21:59:24.892091 4457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:24.893491 4463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220755250ef641deab48e701d4f9c160. Latest consensus state: current_term: 1 leader_uuid: "220755250ef641deab48e701d4f9c160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } } }
I20260620 21:59:24.893649 4463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:24.894395 4472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:24.893903 4464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "220755250ef641deab48e701d4f9c160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220755250ef641deab48e701d4f9c160" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 38951 } } }
I20260620 21:59:24.894547 4464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:24.897040 4472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:24.902197 4472 catalog_manager.cc:1347] Generated new cluster ID: b4205876139441fca7f3f396e8f78f2e
I20260620 21:59:24.902274 4472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:24.915877 4472 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:24.916831 4472 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:24.925791 4472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: Generated new TSK 0
I20260620 21:59:24.926537 4472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.67.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
Entry for principal kudu/127.17.67.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab.
I20260620 21:59:24.956254 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:25.066285 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:25.066569 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:25.066644 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:25.070631 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:25.070716 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:25.070801 4483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:25.076047 4483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.193.keytab
--principal=kudu/127.17.67.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:25.077070 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:25.078104 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:25.087414 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.087415 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.087415 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:25.087587 4483 server_base.cc:1034] running on GCE node
I20260620 21:59:25.088178 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:25.088881 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:25.090065 4483 hybrid_clock.cc:648] HybridClock initialized: now 1781992765090050 us; error 31 us; skew 500 ppm
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:25.092846 4483 init.cc:377] Logged in from keytab as kudu/127.17.67.193@KRBTEST.COM (short username kudu)
I20260620 21:59:25.093842 4483 webserver.cc:460] Webserver started at http://127.17.67.193:44381/ using document root <none> and password file <none>
I20260620 21:59:25.094401 4483 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:25.094502 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:25.094708 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:25.096724 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "3ddecaaa628a4593867c8be311f5d1a3"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a03ef6f7427ddc84c3867b15603b13e8"
server_key_iv: "33d3f3815763d691a3cf45573c0fcd93"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.097285 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "3ddecaaa628a4593867c8be311f5d1a3"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a03ef6f7427ddc84c3867b15603b13e8"
server_key_iv: "33d3f3815763d691a3cf45573c0fcd93"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.100936 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:59:25.103396 4498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.104647 4483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:59:25.104782 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "3ddecaaa628a4593867c8be311f5d1a3"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a03ef6f7427ddc84c3867b15603b13e8"
server_key_iv: "33d3f3815763d691a3cf45573c0fcd93"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.104894 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:25.129081 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:25.132028 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:25.132241 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:25.132912 4483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:25.133831 4483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:25.133903 4483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.133971 4483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:25.134007 4483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.143419 4483 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:39195
I20260620 21:59:25.143437 4611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:39195 every 8 connection(s)
I20260620 21:59:25.144317 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:25.153209 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4483
I20260620 21:59:25.153376 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:25.153705 17679 external_mini_cluster.cc:1428] Setting key 8a14dcdd6857f6aee9ac513f4a1139c2
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.193@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:25.158195 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.146213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:39471 (local address 127.17.67.254:38951)
0620 21:59:25.146347 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.146351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.147149 (+ 798us) server_negotiation.cc:407] Connection header received
0620 21:59:25.148589 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.148592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.148656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.148726 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.149932 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.150521 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.151600 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.151823 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.151896 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.154531 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:25.154535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:25.154538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:25.154565 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:25.156089 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.156615 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.156620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.156622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.156702 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.157062 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.157065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.157067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.157245 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:25.157354 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.157897 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.158031 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:59:25.158983 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.146546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38951 (local address 127.17.67.193:39471)
0620 21:59:25.146991 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.147034 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.148356 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.148893 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.148900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.149241 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.149802 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.149816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.150632 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.150634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.151500 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.151508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.151945 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.151947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.152082 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.152591 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.152606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.154353 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:25.156225 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.156231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.156245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.156508 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.156804 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.156807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.156809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.156956 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.157352 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:25.157359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:25.157666 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.157857 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.158179 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260620 21:59:25.160241 4612 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38951
I20260620 21:59:25.160491 4612 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:25.160969 4612 heartbeater.cc:511] Master 127.17.67.254:38951 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.67.194@KRBTEST.COM; defaulting to no policy
I20260620 21:59:25.162647 4421 ts_manager.cc:194] Registered new tserver with Master: 3ddecaaa628a4593867c8be311f5d1a3 (127.17.67.193:39195)
I20260620 21:59:25.163887 4421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.193@KRBTEST.COM'} at 127.17.67.193:39471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
Entry for principal kudu/127.17.67.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab.
I20260620 21:59:25.180022 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:25.291050 4617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:25.291281 4617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:25.291322 4617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:25.295588 4617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:25.295642 4617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:25.295701 4617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:25.300884 4617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.194.keytab
--principal=kudu/127.17.67.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:25.301894 4617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:25.302860 4617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:25.312197 4625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.312172 4622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.312168 4623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:25.312767 4617 server_base.cc:1034] running on GCE node
I20260620 21:59:25.313148 4617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:25.313854 4617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:25.315073 4617 hybrid_clock.cc:648] HybridClock initialized: now 1781992765315054 us; error 26 us; skew 500 ppm
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:25.317795 4617 init.cc:377] Logged in from keytab as kudu/127.17.67.194@KRBTEST.COM (short username kudu)
I20260620 21:59:25.318693 4617 webserver.cc:460] Webserver started at http://127.17.67.194:40841/ using document root <none> and password file <none>
I20260620 21:59:25.319242 4617 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:25.319336 4617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:25.319555 4617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:25.321506 4617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "93dcb8213cd04998bace3fb31652252b"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a7e1d71961ca21dad99e8a9d248e526d"
server_key_iv: "33cbbd0d7f6c0742ef063bbd3a512b2c"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.322117 4617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "93dcb8213cd04998bace3fb31652252b"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a7e1d71961ca21dad99e8a9d248e526d"
server_key_iv: "33cbbd0d7f6c0742ef063bbd3a512b2c"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.325734 4617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:25.328106 4632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.329388 4617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:25.329556 4617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "93dcb8213cd04998bace3fb31652252b"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "a7e1d71961ca21dad99e8a9d248e526d"
server_key_iv: "33cbbd0d7f6c0742ef063bbd3a512b2c"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.329752 4617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:25.373322 4617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:25.376544 4617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:25.376754 4617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:25.377368 4617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:25.378273 4617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:25.378350 4617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.378427 4617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:25.378471 4617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.388113 4617 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:39387
I20260620 21:59:25.388130 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:39387 every 8 connection(s)
I20260620 21:59:25.389057 4617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:25.396870 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4617
I20260620 21:59:25.397017 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:25.397360 17679 external_mini_cluster.cc:1428] Setting key 8dcbfd334be00bf0f3b4a0b70ea47847
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.194@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:25.403285 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.391093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:37611 (local address 127.17.67.254:38951)
0620 21:59:25.391231 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.391235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.392127 (+ 892us) server_negotiation.cc:407] Connection header received
0620 21:59:25.393698 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.393701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.393762 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.393870 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.395074 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.395598 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.396742 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.397018 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.397142 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.399688 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:25.399692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:25.399695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:25.399723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:25.401183 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.401763 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.401769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.401770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.401852 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.402203 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.402207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.402208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.402384 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:25.402489 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.402990 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.403120 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:59:25.404104 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.391474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38951 (local address 127.17.67.194:37611)
0620 21:59:25.391966 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.392010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.393398 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.394043 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.394050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.394439 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.394933 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.394946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.395726 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.395730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.396617 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.396626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.397193 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.397197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.397357 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.398047 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.398069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.399528 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:25.401318 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.401324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.401338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.401666 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.401951 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.401954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.401956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.402101 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.402498 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:25.402506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:25.402750 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.402938 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.403250 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260620 21:59:25.405256 4746 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38951
I20260620 21:59:25.405491 4746 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:25.405978 4746 heartbeater.cc:511] Master 127.17.67.254:38951 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.67.195@KRBTEST.COM; defaulting to no policy
I20260620 21:59:25.406981 4421 ts_manager.cc:194] Registered new tserver with Master: 93dcb8213cd04998bace3fb31652252b (127.17.67.194:39387)
I20260620 21:59:25.407583 4421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.194@KRBTEST.COM'} at 127.17.67.194:37611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
Entry for principal kudu/127.17.67.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab.
I20260620 21:59:25.422608 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:25.530690 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:25.531002 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:25.531082 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:25.535046 4751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:25.535143 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:25.535234 4751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:25.540366 4751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.195.keytab
--principal=kudu/127.17.67.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:38951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:25.541405 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:25.542690 4751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:25.552799 4757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.552822 4756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:25.552793 4759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:25.553246 4751 server_base.cc:1034] running on GCE node
I20260620 21:59:25.553710 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:25.554340 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:25.555546 4751 hybrid_clock.cc:648] HybridClock initialized: now 1781992765555522 us; error 45 us; skew 500 ppm
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:25.558270 4751 init.cc:377] Logged in from keytab as kudu/127.17.67.195@KRBTEST.COM (short username kudu)
I20260620 21:59:25.559055 4751 webserver.cc:460] Webserver started at http://127.17.67.195:43047/ using document root <none> and password file <none>
I20260620 21:59:25.559561 4751 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:25.559617 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:25.559769 4751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:25.561764 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "9b5d2597c1784d8d9b11ec5c8f54f195"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "59fbcf440d0bc5a7c50dd5abe37bf9f2"
server_key_iv: "525bcf976dd70bccc9c9896326390f3a"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.562269 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "9b5d2597c1784d8d9b11ec5c8f54f195"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "59fbcf440d0bc5a7c50dd5abe37bf9f2"
server_key_iv: "525bcf976dd70bccc9c9896326390f3a"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.565866 4751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:59:25.568149 4766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.569335 4751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:25.569427 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "9b5d2597c1784d8d9b11ec5c8f54f195"
format_stamp: "Formatted at 2026-06-20 21:59:25 on dist-test-slave-cx8d"
server_key: "59fbcf440d0bc5a7c50dd5abe37bf9f2"
server_key_iv: "525bcf976dd70bccc9c9896326390f3a"
server_key_version: "encryptionkey@0"
I20260620 21:59:25.569507 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:25.588996 4751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:25.592156 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:25.592309 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:25.592896 4751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:25.593726 4751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:25.593775 4751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.593813 4751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:25.593830 4751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:25.602489 4751 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:43739
I20260620 21:59:25.602514 4879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:43739 every 8 connection(s)
I20260620 21:59:25.603371 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:25.608623 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4751
I20260620 21:59:25.608773 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:25.609114 17679 external_mini_cluster.cc:1428] Setting key 73d1e56e2721ef8def27ff81c951d3d8
Jun 20 21:59:25 dist-test-slave-cx8d krb5kdc[4376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992765, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.195@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:25.617318 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.605132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:40145 (local address 127.17.67.254:38951)
0620 21:59:25.605292 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.605296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.606149 (+ 853us) server_negotiation.cc:407] Connection header received
0620 21:59:25.607405 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.607408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.607455 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.607532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.608965 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.609665 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.611304 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.611576 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.611654 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.614160 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:25.614164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:25.614171 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:25.614196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:25.615352 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.615850 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.615855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.615857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.615933 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.616277 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.616282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.616284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.616457 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:25.616575 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.617040 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.617156 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:59:25.618223 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.605481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:38951 (local address 127.17.67.195:40145)
0620 21:59:25.606003 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.606045 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.607214 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.607694 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.607700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.608116 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:25.608830 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.608850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.609814 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.609817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.611187 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.611198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.611722 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.611726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.611914 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.612539 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:25.612555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:25.613993 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:25.615474 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.615480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.615495 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.615745 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.616044 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:25.616047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:25.616049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:25.616180 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:25.616573 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:25.616580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:25.616816 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.617001 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.617311 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":114,"threads_started":1}
I20260620 21:59:25.619252 4880 heartbeater.cc:347] Connected to a master server at 127.17.67.254:38951
I20260620 21:59:25.619514 4880 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:25.619982 4880 heartbeater.cc:511] Master 127.17.67.254:38951 requested a full tablet report, sending...
I20260620 21:59:25.620940 4421 ts_manager.cc:194] Registered new tserver with Master: 9b5d2597c1784d8d9b11ec5c8f54f195 (127.17.67.195:43739)
I20260620 21:59:25.621503 4421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.67.195@KRBTEST.COM'} at 127.17.67.195:40145
I20260620 21:59:25.623001 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:25.631582 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.625027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.17.67.254:38951)
0620 21:59:25.625138 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.625141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.625322 (+ 181us) server_negotiation.cc:407] Connection header received
0620 21:59:25.625409 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.625411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.625444 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.625501 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:25.626181 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.626667 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.627426 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.627633 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.627703 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.629027 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:25.629030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:25.629032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:25.629048 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:25.630269 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.630645 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.630647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.630648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.630713 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:25.630934 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:25.630936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:25.630937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:25.631122 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:25.631221 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.631387 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.631467 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:59:25.634382 4421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:25.637306 4421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:25.661056 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.655517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:39195 (local address 127.0.0.1:49302)
0620 21:59:25.655763 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.655778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.655882 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.656533 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.656539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.656563 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:25.656713 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.656721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.657998 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.658002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.659084 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.659100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.660387 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.660390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.660549 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.660591 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.660716 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.660873 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":30,"thread_start_us":107,"threads_started":1}
I20260620 21:59:25.661044 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.655713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.17.67.193:39195)
0620 21:59:25.656086 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.656103 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.656125 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:59:25.656193 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.656197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.656378 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.656538 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:25.656848 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.657863 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.659219 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.660049 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.660157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.660309 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.660759 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.660865 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":93,"threads_started":1}
I20260620 21:59:25.661511 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.653844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:43739 (local address 127.0.0.1:57554)
0620 21:59:25.654419 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.654469 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.654598 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.655786 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.655792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.655821 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:25.656016 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.656024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.657648 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.657654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.659332 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.659347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.660409 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.660416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.660632 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.660672 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.660762 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.660888 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":60,"spinlock_wait_cycles":5888,"thread_start_us":106,"threads_started":1}
I20260620 21:59:25.662179 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.653939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.17.67.195:43739)
0620 21:59:25.655295 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.655309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.655329 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:59:25.655389 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.655393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.655609 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.656179 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:25.656312 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.657466 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.659488 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.660281 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.660387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.660522 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.661857 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.661983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":98,"threads_started":1}
I20260620 21:59:25.663997 4546 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 (DEFAULT_TABLE table=test-table [id=4cacef93353f4e34a87990e4270c9d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:25.664949 4546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa16d77aa74801bf98a2088b02f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:25.665148 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 (DEFAULT_TABLE table=test-table [id=4cacef93353f4e34a87990e4270c9d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:25.666152 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa16d77aa74801bf98a2088b02f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:25.670693 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.654008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39387 (local address 127.0.0.1:52122)
0620 21:59:25.654886 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.654898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.654976 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.666267 (+ 11291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.666270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.666290 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:25.666391 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.666397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.667949 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.667953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.669097 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.669108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.670286 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.670289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.670402 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.670431 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.670484 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.670558 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":107,"threads_started":1}
I20260620 21:59:25.672611 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.654694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.17.67.194:39387)
0620 21:59:25.665788 (+ 11094us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.665800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.665825 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:59:25.665902 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.665910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.666099 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.666548 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:25.666703 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.667794 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.669249 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.670143 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.671130 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.672332 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.672385 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.672472 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10977,"thread_start_us":127,"threads_started":1}
I20260620 21:59:25.673337 4897 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: Bootstrap starting.
I20260620 21:59:25.675720 4897 tablet_bootstrap.cc:654] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:25.675750 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 (DEFAULT_TABLE table=test-table [id=4cacef93353f4e34a87990e4270c9d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:25.676592 4897 log.cc:824] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:25.676805 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa16d77aa74801bf98a2088b02f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:25.677336 4898 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Bootstrap starting.
I20260620 21:59:25.678874 4897 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: No bootstrap required, opened a new log
I20260620 21:59:25.679075 4897 ts_tablet_manager.cc:1397] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:59:25.680871 4898 tablet_bootstrap.cc:654] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:25.681699 4898 log.cc:824] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:25.683686 4898 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: No bootstrap required, opened a new log
I20260620 21:59:25.683873 4898 ts_tablet_manager.cc:1397] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 21:59:25.684922 4901 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Bootstrap starting.
I20260620 21:59:25.685217 4897 raft_consensus.cc:348] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.685634 4897 raft_consensus.cc:374] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:25.685741 4897 raft_consensus.cc:729] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddecaaa628a4593867c8be311f5d1a3, State: Initialized, Role: FOLLOWER
I20260620 21:59:25.686321 4897 consensus_queue.cc:260] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.687037 4901 tablet_bootstrap.cc:654] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:25.687601 4897 ts_tablet_manager.cc:1428] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260620 21:59:25.688184 4612 heartbeater.cc:503] Master 127.17.67.254:38951 was elected leader, sending a full tablet report...
I20260620 21:59:25.688396 4901 log.cc:824] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:25.688825 4898 raft_consensus.cc:348] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.689100 4898 raft_consensus.cc:374] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:25.689150 4898 raft_consensus.cc:729] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5d2597c1784d8d9b11ec5c8f54f195, State: Initialized, Role: FOLLOWER
I20260620 21:59:25.689589 4898 consensus_queue.cc:260] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.690246 4880 heartbeater.cc:503] Master 127.17.67.254:38951 was elected leader, sending a full tablet report...
I20260620 21:59:25.690523 4898 ts_tablet_manager.cc:1428] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260620 21:59:25.691097 4901 tablet_bootstrap.cc:492] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: No bootstrap required, opened a new log
I20260620 21:59:25.691277 4901 ts_tablet_manager.cc:1397] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:25.695308 4901 raft_consensus.cc:348] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.695497 4901 raft_consensus.cc:374] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:25.695572 4901 raft_consensus.cc:729] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dcb8213cd04998bace3fb31652252b, State: Initialized, Role: FOLLOWER
I20260620 21:59:25.695936 4901 consensus_queue.cc:260] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.696552 4746 heartbeater.cc:503] Master 127.17.67.254:38951 was elected leader, sending a full tablet report...
I20260620 21:59:25.696926 4901 ts_tablet_manager.cc:1428] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:59:25.776895 4902 raft_consensus.cc:482] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:25.777130 4902 raft_consensus.cc:504] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.778400 4902 leader_election.cc:288] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5d2597c1784d8d9b11ec5c8f54f195 (127.17.67.195:43739), 93dcb8213cd04998bace3fb31652252b (127.17.67.194:39387)
I20260620 21:59:25.782728 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.778754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:57779 (local address 127.17.67.195:43739)
0620 21:59:25.779147 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.779154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.779173 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:59:25.779233 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.779239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.779302 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.779390 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:25.779801 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.780529 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.781747 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.782143 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.782280 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.782337 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.782536 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.782607 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260620 21:59:25.782752 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.778586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:43739 (local address 127.17.67.193:57779)
0620 21:59:25.778951 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.778967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.779095 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.779485 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.779488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.779511 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:25.779654 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.779665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.780669 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.780673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.781617 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.781633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.782269 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.782272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.782383 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.782411 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.782496 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.782572 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"spinlock_wait_cycles":10368,"thread_start_us":132,"threads_started":1}
I20260620 21:59:25.783408 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa16d77aa74801bf98a2088b02f8e5" candidate_uuid: "3ddecaaa628a4593867c8be311f5d1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" is_pre_election: true
I20260620 21:59:25.783748 4834 raft_consensus.cc:2454] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddecaaa628a4593867c8be311f5d1a3 in term 0.
I20260620 21:59:25.784039 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.778681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:39387 (local address 127.17.67.193:53127)
0620 21:59:25.778951 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.778967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:25.779095 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:25.779749 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:25.779754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:25.779781 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:25.779964 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.779973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.780906 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.780912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.782471 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:25.782486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.783574 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.783578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:25.783673 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.783695 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:59:25.783813 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:59:25.783882 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":27136,"thread_start_us":129,"threads_started":1}
I20260620 21:59:25.784209 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.779278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:53127 (local address 127.17.67.194:39387)
0620 21:59:25.779504 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.779508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.779521 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:25.779571 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.779573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.779623 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.779711 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:25.780108 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.780793 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.782635 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.783433 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.783584 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.783649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.783937 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.784035 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 21:59:25.784263 4501 leader_election.cc:302] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddecaaa628a4593867c8be311f5d1a3, 9b5d2597c1784d8d9b11ec5c8f54f195; no voters:
I20260620 21:59:25.784611 4902 raft_consensus.cc:2781] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:25.784698 4902 raft_consensus.cc:482] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:25.784747 4902 raft_consensus.cc:3037] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:25.784775 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa16d77aa74801bf98a2088b02f8e5" candidate_uuid: "3ddecaaa628a4593867c8be311f5d1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dcb8213cd04998bace3fb31652252b" is_pre_election: true
I20260620 21:59:25.785059 4700 raft_consensus.cc:2454] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddecaaa628a4593867c8be311f5d1a3 in term 0.
I20260620 21:59:25.786051 4902 raft_consensus.cc:504] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.786442 4902 leader_election.cc:288] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5d2597c1784d8d9b11ec5c8f54f195 (127.17.67.195:43739), 93dcb8213cd04998bace3fb31652252b (127.17.67.194:39387)
I20260620 21:59:25.786757 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa16d77aa74801bf98a2088b02f8e5" candidate_uuid: "3ddecaaa628a4593867c8be311f5d1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195"
I20260620 21:59:25.786756 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa16d77aa74801bf98a2088b02f8e5" candidate_uuid: "3ddecaaa628a4593867c8be311f5d1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dcb8213cd04998bace3fb31652252b"
I20260620 21:59:25.786878 4834 raft_consensus.cc:3037] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:25.786878 4700 raft_consensus.cc:3037] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:25.788152 4834 raft_consensus.cc:2454] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddecaaa628a4593867c8be311f5d1a3 in term 1.
I20260620 21:59:25.788152 4700 raft_consensus.cc:2454] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddecaaa628a4593867c8be311f5d1a3 in term 1.
I20260620 21:59:25.788496 4501 leader_election.cc:302] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddecaaa628a4593867c8be311f5d1a3, 9b5d2597c1784d8d9b11ec5c8f54f195; no voters:
I20260620 21:59:25.788690 4902 raft_consensus.cc:2781] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:25.788959 4902 raft_consensus.cc:686] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ddecaaa628a4593867c8be311f5d1a3, State: Running, Role: LEADER
I20260620 21:59:25.789280 4902 consensus_queue.cc:237] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } }
I20260620 21:59:25.792829 4421 catalog_manager.cc:5495] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ddecaaa628a4593867c8be311f5d1a3 (127.17.67.193). New cstate: current_term: 1 leader_uuid: "3ddecaaa628a4593867c8be311f5d1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ddecaaa628a4593867c8be311f5d1a3" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 39195 } health_report { overall_health: HEALTHY } } }
I20260620 21:59:25.807366 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.803720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.17.67.193:39195)
0620 21:59:25.803860 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:25.803865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:25.803879 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:25.804007 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:25.804010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:25.804063 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:25.804175 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:25.804437 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.804934 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.805937 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:25.806160 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:25.806270 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:25.806588 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:25.806995 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:25.807133 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:25.807169 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 21:59:25.807224 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:59:25.813344 4834 raft_consensus.cc:1261] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 1 FOLLOWER]: Refusing update from remote peer 3ddecaaa628a4593867c8be311f5d1a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:25.813344 4700 raft_consensus.cc:1261] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 1 FOLLOWER]: Refusing update from remote peer 3ddecaaa628a4593867c8be311f5d1a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:25.814159 4908 consensus_queue.cc:1035] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dcb8213cd04998bace3fb31652252b" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:25.814383 4902 consensus_queue.cc:1035] T 42fa16d77aa74801bf98a2088b02f8e5 P 3ddecaaa628a4593867c8be311f5d1a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5d2597c1784d8d9b11ec5c8f54f195" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:25.822223 4913 mvcc.cc:205] Tried to move back new op lower bound from 7299042368764096512 to 7299042368675643392. Current Snapshot: MvccSnapshot[applied={T|T < 7299042368764096512}]
I20260620 21:59:25.822234 4911 mvcc.cc:205] Tried to move back new op lower bound from 7299042368764096512 to 7299042368675643392. Current Snapshot: MvccSnapshot[applied={T|T < 7299042368764096512}]
I20260620 21:59:25.826710 4912 mvcc.cc:205] Tried to move back new op lower bound from 7299042368764096512 to 7299042368675643392. Current Snapshot: MvccSnapshot[applied={T|T < 7299042368764096512}]
I20260620 21:59:25.831666 4420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:25.831838 4420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:25.834491 4420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 220755250ef641deab48e701d4f9c160: Sending DeleteTablet for 3 replicas of tablet 42fa16d77aa74801bf98a2088b02f8e5
I20260620 21:59:25.835254 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:57554
I20260620 21:59:25.835491 4680 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:52122
I20260620 21:59:25.835803 4921 tablet_replica.cc:330] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: stopping tablet replica
I20260620 21:59:25.835914 4546 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa16d77aa74801bf98a2088b02f8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:25 UTC) from {username='kudu', principal='kudu/127.17.67.254@KRBTEST.COM'} at 127.0.0.1:49302
I20260620 21:59:25.836236 4921 raft_consensus.cc:2229] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:25.836628 4921 raft_consensus.cc:2258] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:25.837042 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4483
I20260620 21:59:25.838522 4921 ts_tablet_manager.cc:1905] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:25.840525 4922 tablet_replica.cc:330] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: stopping tablet replica
I20260620 21:59:25.840801 4922 raft_consensus.cc:2229] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:25.841109 4922 raft_consensus.cc:2258] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:25.841228 4921 ts_tablet_manager.cc:1918] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:25.841316 4921 log.cc:1196] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/42fa16d77aa74801bf98a2088b02f8e5
I20260620 21:59:25.841650 4921 ts_tablet_manager.cc:1939] T 42fa16d77aa74801bf98a2088b02f8e5 P 9b5d2597c1784d8d9b11ec5c8f54f195: Deleting consensus metadata
I20260620 21:59:25.842540 4922 ts_tablet_manager.cc:1905] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:25.842905 4407 catalog_manager.cc:4841] TS 9b5d2597c1784d8d9b11ec5c8f54f195 (127.17.67.195:43739): tablet 42fa16d77aa74801bf98a2088b02f8e5 (table test-table [id=4cacef93353f4e34a87990e4270c9d68]) successfully deleted
I20260620 21:59:25.844630 4922 ts_tablet_manager.cc:1918] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:25.844746 4922 log.cc:1196] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/42fa16d77aa74801bf98a2088b02f8e5
I20260620 21:59:25.845093 4922 ts_tablet_manager.cc:1939] T 42fa16d77aa74801bf98a2088b02f8e5 P 93dcb8213cd04998bace3fb31652252b: Deleting consensus metadata
W20260620 21:59:25.846428 4406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:39195 (error 108)
I20260620 21:59:25.846472 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4617
I20260620 21:59:25.847965 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:25.847696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:39195 (local address 127.0.0.1:49322)
0620 21:59:25.847820 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:25.847885 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:59:25.848124 4406 catalog_manager.cc:4584] TS 3ddecaaa628a4593867c8be311f5d1a3 (127.17.67.193:39195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42fa16d77aa74801bf98a2088b02f8e5: Network error: Client connection negotiation failed: client connection to 127.17.67.193:39195: connect: Connection refused (error 111)
I20260620 21:59:25.848178 4408 catalog_manager.cc:4841] TS 93dcb8213cd04998bace3fb31652252b (127.17.67.194:39387): tablet 42fa16d77aa74801bf98a2088b02f8e5 (table test-table [id=4cacef93353f4e34a87990e4270c9d68]) successfully deleted
I20260620 21:59:25.852324 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4751
I20260620 21:59:25.857815 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4389
2026-06-20T21:59:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2375 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T21:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:25Z Disabled control of system clock
I20260620 21:59:25.888655 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39307
--webserver_interface=127.17.67.254
--webserver_port=0
--builtin_ntp_servers=127.17.67.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:25.999429 4928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:25.999701 4928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:25.999737 4928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:26.003839 4928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:26.003897 4928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:26.003913 4928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:26.003930 4928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:26.003983 4928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:26.008749 4928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:26.009887 4928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:26.011330 4928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:26.019737 4933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.019783 4934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.019759 4936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.020200 4928 server_base.cc:1034] running on GCE node
I20260620 21:59:26.020731 4928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:26.021888 4928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:26.023061 4928 hybrid_clock.cc:648] HybridClock initialized: now 1781992766023045 us; error 27 us; skew 500 ppm
I20260620 21:59:26.025058 4928 webserver.cc:460] Webserver started at http://127.17.67.254:40023/ using document root <none> and password file <none>
I20260620 21:59:26.025705 4928 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:26.025774 4928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:26.025992 4928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:26.028031 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "86271e0b75d342a5bb61024cbe182981"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "abcc4a1c1d33d85afdb7cb05e6f0fdd7"
server_key_iv: "bd6986b1c47aee291abf53c3d0c5dce5"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.028642 4928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "86271e0b75d342a5bb61024cbe182981"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "abcc4a1c1d33d85afdb7cb05e6f0fdd7"
server_key_iv: "bd6986b1c47aee291abf53c3d0c5dce5"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.032366 4928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 21:59:26.034853 4942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.036046 4928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:26.036177 4928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "86271e0b75d342a5bb61024cbe182981"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "abcc4a1c1d33d85afdb7cb05e6f0fdd7"
server_key_iv: "bd6986b1c47aee291abf53c3d0c5dce5"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.036289 4928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:26.069123 4928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:26.069903 4928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:26.070119 4928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:26.077093 4928 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39307
I20260620 21:59:26.077101 4994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39307 every 8 connection(s)
I20260620 21:59:26.078076 4928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:26.081178 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:26.084702 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 4928
I20260620 21:59:26.084837 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:26.085175 17679 external_mini_cluster.cc:1428] Setting key 81e660363719f270d79de12fccdad7fd
I20260620 21:59:26.087034 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: Bootstrap starting.
I20260620 21:59:26.089291 4995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:26.090077 4995 log.cc:824] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:26.091848 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: No bootstrap required, opened a new log
I20260620 21:59:26.094517 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.086652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.17.67.254:39307)
0620 21:59:26.087323 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.087339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.087373 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:59:26.088008 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.088023 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.088422 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.088767 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.089314 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.090444 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.091341 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.091802 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.091988 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.092842 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:26.092847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:26.092859 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:26.093242 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:26.093294 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:26.093314 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:26.093428 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.093587 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.093836 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260620 21:59:26.095417 4995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } }
I20260620 21:59:26.095654 4995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:26.095713 4995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86271e0b75d342a5bb61024cbe182981, State: Initialized, Role: FOLLOWER
I20260620 21:59:26.096115 4995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } }
I20260620 21:59:26.096226 4995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:26.096310 4995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:26.096416 4995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:26.097261 4995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } }
I20260620 21:59:26.097625 4995 leader_election.cc:302] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86271e0b75d342a5bb61024cbe182981; no voters:
I20260620 21:59:26.097915 4995 leader_election.cc:288] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:26.098050 5000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:26.098308 5000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [term 1 LEADER]: Becoming Leader. State: Replica: 86271e0b75d342a5bb61024cbe182981, State: Running, Role: LEADER
I20260620 21:59:26.098580 5000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } }
I20260620 21:59:26.099092 4995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:26.100162 5001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86271e0b75d342a5bb61024cbe182981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } } }
I20260620 21:59:26.100275 5001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:26.100646 5009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:26.100612 5002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86271e0b75d342a5bb61024cbe182981. Latest consensus state: current_term: 1 leader_uuid: "86271e0b75d342a5bb61024cbe182981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86271e0b75d342a5bb61024cbe182981" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39307 } } }
I20260620 21:59:26.101300 5002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:26.103847 5009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:26.109678 5009 catalog_manager.cc:1347] Generated new cluster ID: 444fb65575c642339f55c0ef44e76a36
I20260620 21:59:26.109745 5009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:26.141038 5009 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:26.142359 5009 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:26.148638 5009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: Generated new TSK 0
I20260620 21:59:26.149430 5009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:26.154639 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.193:0
--local_ip_for_outbound_sockets=127.17.67.193
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--builtin_ntp_servers=127.17.67.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:59:26.267892 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:26.268149 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:26.268187 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:26.272373 5019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:26.272429 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:26.272483 5019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.193
I20260620 21:59:26.277117 5019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.67.193
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.193
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:26.278213 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:26.279585 5019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:26.289002 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.289078 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.289127 5019 server_base.cc:1034] running on GCE node
W20260620 21:59:26.289083 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.289923 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:26.290652 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:26.291842 5019 hybrid_clock.cc:648] HybridClock initialized: now 1781992766291807 us; error 44 us; skew 500 ppm
I20260620 21:59:26.293804 5019 webserver.cc:460] Webserver started at http://127.17.67.193:46417/ using document root <none> and password file <none>
I20260620 21:59:26.294366 5019 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:26.294427 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:26.294654 5019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:26.296680 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data/instance:
uuid: "967fde52f3a1436394c2d47b466498c8"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "ac8df6e6166fc7fc41506ece1556640a"
server_key_iv: "6bb8722abfa0f00b33d7152713f91a15"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.297251 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance:
uuid: "967fde52f3a1436394c2d47b466498c8"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "ac8df6e6166fc7fc41506ece1556640a"
server_key_iv: "6bb8722abfa0f00b33d7152713f91a15"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.301173 5019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 21:59:26.303730 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.304970 5019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:26.305117 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal
uuid: "967fde52f3a1436394c2d47b466498c8"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "ac8df6e6166fc7fc41506ece1556640a"
server_key_iv: "6bb8722abfa0f00b33d7152713f91a15"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.305234 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:26.325729 5019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:26.326455 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:26.326671 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:26.327301 5019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:26.328357 5019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:26.328444 5019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.328519 5019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:26.328568 5019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.338150 5019 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.193:40947
I20260620 21:59:26.338178 5146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.193:40947 every 8 connection(s)
I20260620 21:59:26.339085 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/data/info.pb
I20260620 21:59:26.341439 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5019
I20260620 21:59:26.341609 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-0/wal/instance
I20260620 21:59:26.341914 17679 external_mini_cluster.cc:1428] Setting key 86a7dccc3c45edd66b7a44e43f7c4e20
I20260620 21:59:26.343959 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.194:0
--local_ip_for_outbound_sockets=127.17.67.194
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--builtin_ntp_servers=127.17.67.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:26.350320 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.341198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.193:48839 (local address 127.17.67.254:39307)
0620 21:59:26.341376 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.341382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.342767 (+ 1385us) server_negotiation.cc:407] Connection header received
0620 21:59:26.344371 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.344378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.344450 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.344558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.345803 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.346777 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.348048 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.348329 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.348506 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.349777 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:26.349781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:26.349784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:26.349798 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:26.349826 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:26.349835 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:26.349923 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.350106 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.350171 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:59:26.351213 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.341805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39307 (local address 127.17.67.193:48839)
0620 21:59:26.342603 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.342658 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.344111 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.344751 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.344762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.344882 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.345625 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.345649 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.346928 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.346931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.347899 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.347909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.348543 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.348547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.348733 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.349427 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:26.349429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.349471 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:26.349538 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:26.349925 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:26.349938 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:26.349946 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.350090 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.350388 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":194,"threads_started":1}
I20260620 21:59:26.352536 5147 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39307
I20260620 21:59:26.352835 5147 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:26.353615 5147 heartbeater.cc:511] Master 127.17.67.254:39307 requested a full tablet report, sending...
I20260620 21:59:26.355496 4959 ts_manager.cc:194] Registered new tserver with Master: 967fde52f3a1436394c2d47b466498c8 (127.17.67.193:40947)
I20260620 21:59:26.357225 4959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.193:48839
W20260620 21:59:26.466971 5150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:26.467224 5150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:26.467262 5150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:26.471236 5150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:26.471290 5150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:26.471343 5150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.194
I20260620 21:59:26.475979 5150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.67.194
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.194
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:26.476956 5150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:26.478331 5150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:26.487906 5158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.487893 5156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.487888 5155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.488363 5150 server_base.cc:1034] running on GCE node
I20260620 21:59:26.488763 5150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:26.489398 5150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:26.490559 5150 hybrid_clock.cc:648] HybridClock initialized: now 1781992766490499 us; error 92 us; skew 500 ppm
I20260620 21:59:26.497188 5150 webserver.cc:460] Webserver started at http://127.17.67.194:44007/ using document root <none> and password file <none>
I20260620 21:59:26.497852 5150 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:26.497918 5150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:26.498145 5150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:26.500404 5150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data/instance:
uuid: "d82592618ef14363b71ed6e177b5de63"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "36c09983afce44ff4865d53a716b4bab"
server_key_iv: "0b7943e5b9f759603a25bef815c90e1a"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.501045 5150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance:
uuid: "d82592618ef14363b71ed6e177b5de63"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "36c09983afce44ff4865d53a716b4bab"
server_key_iv: "0b7943e5b9f759603a25bef815c90e1a"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.504698 5150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:59:26.507092 5164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.508263 5150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:26.508396 5150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal
uuid: "d82592618ef14363b71ed6e177b5de63"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "36c09983afce44ff4865d53a716b4bab"
server_key_iv: "0b7943e5b9f759603a25bef815c90e1a"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.508514 5150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:26.526497 5150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:26.527163 5150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:26.527370 5150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:26.527967 5150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:26.528919 5150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:26.528966 5150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.529008 5150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:26.529065 5150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.537949 5150 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.194:42399
I20260620 21:59:26.538121 5277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.194:42399 every 8 connection(s)
I20260620 21:59:26.538892 5150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/data/info.pb
I20260620 21:59:26.541098 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5150
I20260620 21:59:26.541241 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal/instance
I20260620 21:59:26.541599 17679 external_mini_cluster.cc:1428] Setting key 1ceab3a985e46ed5624fff105b416181
I20260620 21:59:26.543814 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.67.195:0
--local_ip_for_outbound_sockets=127.17.67.195
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--builtin_ntp_servers=127.17.67.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:59:26.550709 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.540855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.194:57463 (local address 127.17.67.254:39307)
0620 21:59:26.541025 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.541031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.542312 (+ 1281us) server_negotiation.cc:407] Connection header received
0620 21:59:26.544081 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.544087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.544151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.544209 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.545622 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.546504 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.548357 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.548577 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.548662 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.550175 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:26.550178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:26.550180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:26.550193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:26.550212 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:26.550220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:26.550294 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.550530 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.550590 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:59:26.552086 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.541343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39307 (local address 127.17.67.194:57463)
0620 21:59:26.542128 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.542192 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.543816 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.544630 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.544642 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.544754 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.545448 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.545463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.546653 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.546660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.548212 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.548227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.548753 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.548760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.549002 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.549791 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:26.549795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.549860 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:26.549963 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:26.550320 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:26.550332 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:26.550340 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.550503 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.550860 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":168,"threads_started":1}
I20260620 21:59:26.553442 5278 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39307
I20260620 21:59:26.553741 5278 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:26.554317 5278 heartbeater.cc:511] Master 127.17.67.254:39307 requested a full tablet report, sending...
I20260620 21:59:26.555444 4959 ts_manager.cc:194] Registered new tserver with Master: d82592618ef14363b71ed6e177b5de63 (127.17.67.194:42399)
I20260620 21:59:26.556041 4959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.194:57463
W20260620 21:59:26.658238 5281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:26.658548 5281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:26.658656 5281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:26.662796 5281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:26.662882 5281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:26.662966 5281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.67.195
I20260620 21:59:26.667824 5281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.67.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.67.195
--webserver_port=0
--tserver_master_addrs=127.17.67.254:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.67.195
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:26.668954 5281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:26.670461 5281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:26.680056 5286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:26.680164 5289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.680284 5281 server_base.cc:1034] running on GCE node
W20260620 21:59:26.680056 5287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:26.680796 5281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:26.681738 5281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:26.682943 5281 hybrid_clock.cc:648] HybridClock initialized: now 1781992766682908 us; error 47 us; skew 500 ppm
I20260620 21:59:26.684948 5281 webserver.cc:460] Webserver started at http://127.17.67.195:35825/ using document root <none> and password file <none>
I20260620 21:59:26.685606 5281 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:26.685709 5281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:26.685940 5281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:26.688120 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data/instance:
uuid: "4813e6473f0141c5b8e4510523bb227f"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "e8665169bc2015ebd448d8e82f31168f"
server_key_iv: "df91f99b6b8f8f482831b63b96c3c27e"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.688711 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance:
uuid: "4813e6473f0141c5b8e4510523bb227f"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "e8665169bc2015ebd448d8e82f31168f"
server_key_iv: "df91f99b6b8f8f482831b63b96c3c27e"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.692625 5281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:59:26.695070 5295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.696259 5281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:26.696393 5281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal
uuid: "4813e6473f0141c5b8e4510523bb227f"
format_stamp: "Formatted at 2026-06-20 21:59:26 on dist-test-slave-cx8d"
server_key: "e8665169bc2015ebd448d8e82f31168f"
server_key_iv: "df91f99b6b8f8f482831b63b96c3c27e"
server_key_version: "encryptionkey@0"
I20260620 21:59:26.696530 5281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:26.714461 5281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:26.715150 5281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:26.715350 5281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:26.715950 5281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:59:26.716907 5281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:59:26.716990 5281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.717059 5281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:59:26.717093 5281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:26.726301 5281 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.195:32899
I20260620 21:59:26.726325 5408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.195:32899 every 8 connection(s)
I20260620 21:59:26.727286 5281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/data/info.pb
I20260620 21:59:26.731081 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5281
I20260620 21:59:26.731174 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal/instance
I20260620 21:59:26.731480 17679 external_mini_cluster.cc:1428] Setting key c24c7b43960a3fc1fe62f2c2051b3ca5
I20260620 21:59:26.736680 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.729137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:56029 (local address 127.17.67.254:39307)
0620 21:59:26.729294 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.729298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.730529 (+ 1231us) server_negotiation.cc:407] Connection header received
0620 21:59:26.731943 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.731946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.732006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.732076 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.733122 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.733854 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.734966 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.735179 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.735261 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.736304 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:26.736306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:26.736308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:26.736319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:26.736334 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:26.736340 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:26.736394 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.736527 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.736572 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:59:26.737515 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.729656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39307 (local address 127.17.67.195:56029)
0620 21:59:26.730325 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.730365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.731734 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.732268 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.732294 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.732401 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:26.732986 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.733001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.733967 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.733970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.734848 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.734857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.735314 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.735317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.735453 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.736081 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:59:26.736083 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:26.736121 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:59:26.736182 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:26.736410 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:26.736416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:26.736420 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.736514 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.736747 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260620 21:59:26.738649 5409 heartbeater.cc:347] Connected to a master server at 127.17.67.254:39307
I20260620 21:59:26.738883 5409 heartbeater.cc:464] Registering TS with master...
I20260620 21:59:26.739415 5409 heartbeater.cc:511] Master 127.17.67.254:39307 requested a full tablet report, sending...
I20260620 21:59:26.740417 4959 ts_manager.cc:194] Registered new tserver with Master: 4813e6473f0141c5b8e4510523bb227f (127.17.67.195:32899)
I20260620 21:59:26.741032 4959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.67.195:56029
I20260620 21:59:26.745343 17679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:59:26.750883 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.747352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.17.67.254:39307)
0620 21:59:26.747491 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.747495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.747657 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:59:26.747755 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.747757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.747794 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.747855 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:26.748080 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.748573 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.749365 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.749660 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.749737 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.750471 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:26.750473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:26.750475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:26.750486 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:26.750502 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:26.750538 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:26.750595 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.750672 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.750715 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:59:26.753501 4959 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:59:26.756474 4959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:59:26.770787 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.765914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.17.67.193:40947)
0620 21:59:26.766500 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.766511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.766531 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:59:26.766605 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.766610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.766778 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.766930 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:26.767094 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.768223 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.769599 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770137 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.770265 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.770479 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.770535 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.770625 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":146,"threads_started":1}
I20260620 21:59:26.770782 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.765504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40947 (local address 127.0.0.1:59768)
0620 21:59:26.765965 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.765978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.766061 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.766926 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.766931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.766948 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:26.767019 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.767023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.768373 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.768380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.769450 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.769459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770265 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.770269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.770373 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.770409 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.770510 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.770625 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":12,"thread_start_us":115,"threads_started":1}
I20260620 21:59:26.770978 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.764727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:42399 (local address 127.0.0.1:58676)
0620 21:59:26.765282 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.765331 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.765451 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.766616 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.766625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.766656 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:26.766813 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.766819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.767949 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.767956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.769476 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.769486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770563 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.770566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.770649 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.770671 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.770720 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.770790 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":3,"spinlock_wait_cycles":3840,"thread_start_us":83,"threads_started":1}
I20260620 21:59:26.771152 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.764900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.195:32899 (local address 127.0.0.1:34680)
0620 21:59:26.765817 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.765830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.765922 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.766614 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.766619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.766654 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:26.766821 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.766834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.768336 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.768339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.769533 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.769543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770777 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.770780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.770868 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.770889 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.770950 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.771017 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":108,"spinlock_wait_cycles":3328,"thread_start_us":124,"threads_started":1}
I20260620 21:59:26.771660 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.765065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.17.67.195:32899)
0620 21:59:26.765762 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.765774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.765912 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:59:26.766050 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.766056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.766275 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.766435 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:26.766957 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.768196 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.769677 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770426 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.770545 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.770722 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.771384 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.771483 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":85,"threads_started":1}
I20260620 21:59:26.771901 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.765420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.17.67.194:42399)
0620 21:59:26.766171 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.766182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.766206 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:59:26.766281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.766285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.766474 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.766630 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:26.767118 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.767837 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.769650 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.770426 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.771286 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.771468 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.771510 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.771741 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":141,"threads_started":1}
I20260620 21:59:26.774163 5343 tablet_service.cc:1467] Processing CreateTablet for tablet cbffefa225854a659b9179d0c0633f5c (DEFAULT_TABLE table=test-table [id=38298d55e7b44ad6b4dd80f534732f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:26.774292 5212 tablet_service.cc:1467] Processing CreateTablet for tablet cbffefa225854a659b9179d0c0633f5c (DEFAULT_TABLE table=test-table [id=38298d55e7b44ad6b4dd80f534732f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:26.774291 5081 tablet_service.cc:1467] Processing CreateTablet for tablet cbffefa225854a659b9179d0c0633f5c (DEFAULT_TABLE table=test-table [id=38298d55e7b44ad6b4dd80f534732f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:59:26.775134 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbffefa225854a659b9179d0c0633f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:26.775271 5081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbffefa225854a659b9179d0c0633f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:26.775271 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbffefa225854a659b9179d0c0633f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:26.781100 5426 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Bootstrap starting.
I20260620 21:59:26.782634 5427 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Bootstrap starting.
I20260620 21:59:26.783337 5428 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Bootstrap starting.
I20260620 21:59:26.783438 5426 tablet_bootstrap.cc:654] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:26.784118 5426 log.cc:824] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:26.784724 5427 tablet_bootstrap.cc:654] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:26.785358 5428 tablet_bootstrap.cc:654] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:26.785526 5427 log.cc:824] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:26.786108 5426 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: No bootstrap required, opened a new log
I20260620 21:59:26.786293 5426 ts_tablet_manager.cc:1397] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:59:26.786378 5428 log.cc:824] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:26.787448 5427 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: No bootstrap required, opened a new log
I20260620 21:59:26.787624 5427 ts_tablet_manager.cc:1397] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:59:26.788355 5428 tablet_bootstrap.cc:492] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: No bootstrap required, opened a new log
I20260620 21:59:26.788542 5428 ts_tablet_manager.cc:1397] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:59:26.789922 5426 raft_consensus.cc:348] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.790179 5426 raft_consensus.cc:374] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:26.790230 5426 raft_consensus.cc:729] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4813e6473f0141c5b8e4510523bb227f, State: Initialized, Role: FOLLOWER
I20260620 21:59:26.790810 5426 consensus_queue.cc:260] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.791435 5409 heartbeater.cc:503] Master 127.17.67.254:39307 was elected leader, sending a full tablet report...
I20260620 21:59:26.791846 5427 raft_consensus.cc:348] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.792109 5427 raft_consensus.cc:374] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:26.792160 5427 raft_consensus.cc:729] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82592618ef14363b71ed6e177b5de63, State: Initialized, Role: FOLLOWER
I20260620 21:59:26.792570 5427 consensus_queue.cc:260] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.792717 5428 raft_consensus.cc:348] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.792944 5428 raft_consensus.cc:374] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:26.792994 5428 raft_consensus.cc:729] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967fde52f3a1436394c2d47b466498c8, State: Initialized, Role: FOLLOWER
I20260620 21:59:26.792984 5426 ts_tablet_manager.cc:1428] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:59:26.793503 5427 ts_tablet_manager.cc:1428] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:59:26.793653 5278 heartbeater.cc:503] Master 127.17.67.254:39307 was elected leader, sending a full tablet report...
I20260620 21:59:26.793408 5428 consensus_queue.cc:260] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.794898 5147 heartbeater.cc:503] Master 127.17.67.254:39307 was elected leader, sending a full tablet report...
I20260620 21:59:26.795265 5428 ts_tablet_manager.cc:1428] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 21:59:26.840497 5148 tablet.cc:2369] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:26.859344 5432 raft_consensus.cc:482] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:59:26.859640 5432 raft_consensus.cc:504] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.860790 5432 leader_election.cc:288] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d82592618ef14363b71ed6e177b5de63 (127.17.67.194:42399), 967fde52f3a1436394c2d47b466498c8 (127.17.67.193:40947)
I20260620 21:59:26.864481 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.861245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:48827 (local address 127.17.67.193:40947)
0620 21:59:26.861405 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.861412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.861446 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:59:26.861508 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.861511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.861556 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.861663 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:26.861942 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.862447 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.863528 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.863918 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.864055 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.864107 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.864289 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.864358 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 21:59:26.864482 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.861109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40947 (local address 127.17.67.195:48827)
0620 21:59:26.861246 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.861263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.861374 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.861677 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.861680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.861698 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:26.861818 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.861824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.862575 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.862581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.863400 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.863410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.864055 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.864058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.864156 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.864181 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.864273 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.864346 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":20}
I20260620 21:59:26.865182 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbffefa225854a659b9179d0c0633f5c" candidate_uuid: "4813e6473f0141c5b8e4510523bb227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967fde52f3a1436394c2d47b466498c8" is_pre_election: true
I20260620 21:59:26.865283 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.861246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.67.195:42435 (local address 127.17.67.194:42399)
0620 21:59:26.861426 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.861431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.861903 (+ 472us) server_negotiation.cc:407] Connection header received
0620 21:59:26.862066 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.862069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.862124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.862216 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:59:26.862489 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.863022 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.864149 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.864692 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.864828 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.864886 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.865106 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.865178 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:59:26.865298 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.861109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.194:42399 (local address 127.17.67.195:42435)
0620 21:59:26.861793 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.861814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:26.861946 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:26.862226 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:26.862229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:26.862246 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:59:26.862374 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.862380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.863139 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.863146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.864015 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:26.864025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.864846 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.864850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:26.864957 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.864980 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:59:26.865087 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:59:26.865153 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":296,"thread_start_us":183,"threads_started":1}
I20260620 21:59:26.865625 5101 raft_consensus.cc:2454] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4813e6473f0141c5b8e4510523bb227f in term 0.
I20260620 21:59:26.866112 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbffefa225854a659b9179d0c0633f5c" candidate_uuid: "4813e6473f0141c5b8e4510523bb227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82592618ef14363b71ed6e177b5de63" is_pre_election: true
I20260620 21:59:26.866225 5296 leader_election.cc:302] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4813e6473f0141c5b8e4510523bb227f, 967fde52f3a1436394c2d47b466498c8; no voters:
I20260620 21:59:26.866396 5232 raft_consensus.cc:2454] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4813e6473f0141c5b8e4510523bb227f in term 0.
I20260620 21:59:26.866541 5432 raft_consensus.cc:2781] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:59:26.866639 5432 raft_consensus.cc:482] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:59:26.866689 5432 raft_consensus.cc:3037] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:26.867858 5432 raft_consensus.cc:504] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.868227 5432 leader_election.cc:288] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [CANDIDATE]: Term 1 election: Requested vote from peers d82592618ef14363b71ed6e177b5de63 (127.17.67.194:42399), 967fde52f3a1436394c2d47b466498c8 (127.17.67.193:40947)
I20260620 21:59:26.868618 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbffefa225854a659b9179d0c0633f5c" candidate_uuid: "4813e6473f0141c5b8e4510523bb227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "967fde52f3a1436394c2d47b466498c8"
I20260620 21:59:26.868618 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbffefa225854a659b9179d0c0633f5c" candidate_uuid: "4813e6473f0141c5b8e4510523bb227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82592618ef14363b71ed6e177b5de63"
I20260620 21:59:26.868748 5101 raft_consensus.cc:3037] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:26.868748 5232 raft_consensus.cc:3037] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:26.870179 5101 raft_consensus.cc:2454] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4813e6473f0141c5b8e4510523bb227f in term 1.
I20260620 21:59:26.870179 5232 raft_consensus.cc:2454] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4813e6473f0141c5b8e4510523bb227f in term 1.
I20260620 21:59:26.870580 5299 leader_election.cc:302] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4813e6473f0141c5b8e4510523bb227f, 967fde52f3a1436394c2d47b466498c8, d82592618ef14363b71ed6e177b5de63; no voters:
I20260620 21:59:26.870760 5432 raft_consensus.cc:2781] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:26.871006 5432 raft_consensus.cc:686] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 1 LEADER]: Becoming Leader. State: Replica: 4813e6473f0141c5b8e4510523bb227f, State: Running, Role: LEADER
I20260620 21:59:26.871359 5432 consensus_queue.cc:237] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } }
I20260620 21:59:26.874578 4958 catalog_manager.cc:5495] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4813e6473f0141c5b8e4510523bb227f (127.17.67.195). New cstate: current_term: 1 leader_uuid: "4813e6473f0141c5b8e4510523bb227f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4813e6473f0141c5b8e4510523bb227f" member_type: VOTER last_known_addr { host: "127.17.67.195" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 21:59:26.898976 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.895581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.17.67.195:32899)
0620 21:59:26.895702 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:26.895706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:26.895745 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:59:26.895817 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:26.895819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:26.895873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:26.895957 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:26.896182 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.896673 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.897477 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:26.897782 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:26.897861 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:26.898118 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:26.898558 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:26.898651 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:26.898748 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:59:26.898829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:26.905328 5101 raft_consensus.cc:1261] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 1 FOLLOWER]: Refusing update from remote peer 4813e6473f0141c5b8e4510523bb227f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:26.905334 5232 raft_consensus.cc:1261] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 1 FOLLOWER]: Refusing update from remote peer 4813e6473f0141c5b8e4510523bb227f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:26.906136 5436 consensus_queue.cc:1035] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [LEADER]: Connected to new peer: Peer: permanent_uuid: "967fde52f3a1436394c2d47b466498c8" member_type: VOTER last_known_addr { host: "127.17.67.193" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:26.906365 5432 consensus_queue.cc:1035] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82592618ef14363b71ed6e177b5de63" member_type: VOTER last_known_addr { host: "127.17.67.194" port: 42399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:26.923851 4958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:26.924060 4958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:59:26.926877 4958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86271e0b75d342a5bb61024cbe182981: Sending DeleteTablet for 3 replicas of tablet cbffefa225854a659b9179d0c0633f5c
I20260620 21:59:26.927882 5081 tablet_service.cc:1514] Processing DeleteTablet for tablet cbffefa225854a659b9179d0c0633f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:26 UTC) from {username='slave'} at 127.0.0.1:59768
I20260620 21:59:26.927855 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet cbffefa225854a659b9179d0c0633f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:26 UTC) from {username='slave'} at 127.0.0.1:58676
I20260620 21:59:26.928076 5343 tablet_service.cc:1514] Processing DeleteTablet for tablet cbffefa225854a659b9179d0c0633f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:59:26 UTC) from {username='slave'} at 127.0.0.1:34680
I20260620 21:59:26.928658 5449 tablet_replica.cc:330] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8: stopping tablet replica
I20260620 21:59:26.928879 5451 tablet_replica.cc:330] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: stopping tablet replica
I20260620 21:59:26.929001 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5019
I20260620 21:59:26.929040 5449 raft_consensus.cc:2229] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:26.929390 5449 raft_consensus.cc:2258] T cbffefa225854a659b9179d0c0633f5c P 967fde52f3a1436394c2d47b466498c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:26.929690 5450 tablet_replica.cc:330] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: stopping tablet replica
I20260620 21:59:26.929994 5450 raft_consensus.cc:2229] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:59:26.930166 5451 raft_consensus.cc:2229] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:59:26.930301 5450 raft_consensus.cc:2258] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:26.930722 5451 raft_consensus.cc:2258] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:59:26.932533 5451 ts_tablet_manager.cc:1905] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:26.932713 5450 ts_tablet_manager.cc:1905] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:59:26.935590 5451 ts_tablet_manager.cc:1918] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:26.935583 5450 ts_tablet_manager.cc:1918] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:59:26.935690 5450 log.cc:1196] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-1/wal/wals/cbffefa225854a659b9179d0c0633f5c
I20260620 21:59:26.935817 5451 log.cc:1196] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Deleting WAL directory at /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992513246304-17679-0/minicluster-data/ts-2/wal/wals/cbffefa225854a659b9179d0c0633f5c
I20260620 21:59:26.936036 5450 ts_tablet_manager.cc:1939] T cbffefa225854a659b9179d0c0633f5c P d82592618ef14363b71ed6e177b5de63: Deleting consensus metadata
I20260620 21:59:26.936321 5451 ts_tablet_manager.cc:1939] T cbffefa225854a659b9179d0c0633f5c P 4813e6473f0141c5b8e4510523bb227f: Deleting consensus metadata
I20260620 21:59:26.937448 4943 catalog_manager.cc:4841] TS 4813e6473f0141c5b8e4510523bb227f (127.17.67.195:32899): tablet cbffefa225854a659b9179d0c0633f5c (table test-table [id=38298d55e7b44ad6b4dd80f534732f67]) successfully deleted
I20260620 21:59:26.937454 4946 catalog_manager.cc:4841] TS d82592618ef14363b71ed6e177b5de63 (127.17.67.194:42399): tablet cbffefa225854a659b9179d0c0633f5c (table test-table [id=38298d55e7b44ad6b4dd80f534732f67]) successfully deleted
W20260620 21:59:26.938393 4943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.67.193:40947 (error 108)
I20260620 21:59:26.939291 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5150
I20260620 21:59:26.939561 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:26.939230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.193:40947 (local address 127.0.0.1:59776)
0620 21:59:26.939394 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:26.939476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 21:59:26.940685 4943 catalog_manager.cc:4584] TS 967fde52f3a1436394c2d47b466498c8 (127.17.67.193:40947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbffefa225854a659b9179d0c0633f5c: Network error: Client connection negotiation failed: client connection to 127.17.67.193:40947: connect: Connection refused (error 111)
I20260620 21:59:26.945389 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5281
I20260620 21:59:26.951749 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 4928
2026-06-20T21:59:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1093 ms)
[----------] 6 tests from EncryptionPolicyTest (10577 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:27 dist-test-slave-cx8d krb5kdc[5455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:27 dist-test-slave-cx8d krb5kdc[5455](info): set up 2 sockets
Jun 20 21:59:27 dist-test-slave-cx8d krb5kdc[5455](info): commencing operation
krb5kdc: starting...
W20260620 21:59:27.997656 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:28.118100 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:28.227438 5468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:28.227715 5468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:28.227756 5468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:28.231765 5468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:28.231827 5468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:28.231846 5468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:28.231863 5468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:28.231878 5468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:28.237177 5468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:28.238268 5468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:28.239326 5468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:28.247587 5476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:28.247597 5473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:28.247594 5474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:28.247753 5468 server_base.cc:1034] running on GCE node
I20260620 21:59:28.248553 5468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:28.249729 5468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:28.250913 5468 hybrid_clock.cc:648] HybridClock initialized: now 1781992768250870 us; error 54 us; skew 500 ppm
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:28.253706 5468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:28.254555 5468 webserver.cc:460] Webserver started at http://127.0.0.1:34681/ using document root <none> and password file <none>
I20260620 21:59:28.255107 5468 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:28.255165 5468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:28.255393 5468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:28.257393 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "b0896aa443c34e1da1bdb1439ca9c28f"
format_stamp: "Formatted at 2026-06-20 21:59:28 on dist-test-slave-cx8d"
server_key: "73edbe0ccccc5da6c825f6a5e45c6b6d"
server_key_iv: "020457ab253bee3922fd31ea4d17bbe5"
server_key_version: "encryptionkey@0"
I20260620 21:59:28.258042 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "b0896aa443c34e1da1bdb1439ca9c28f"
format_stamp: "Formatted at 2026-06-20 21:59:28 on dist-test-slave-cx8d"
server_key: "73edbe0ccccc5da6c825f6a5e45c6b6d"
server_key_iv: "020457ab253bee3922fd31ea4d17bbe5"
server_key_version: "encryptionkey@0"
I20260620 21:59:28.261624 5468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:28.263921 5483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:28.265100 5468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:28.265199 5468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "b0896aa443c34e1da1bdb1439ca9c28f"
format_stamp: "Formatted at 2026-06-20 21:59:28 on dist-test-slave-cx8d"
server_key: "73edbe0ccccc5da6c825f6a5e45c6b6d"
server_key_iv: "020457ab253bee3922fd31ea4d17bbe5"
server_key_version: "encryptionkey@0"
I20260620 21:59:28.265329 5468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:28.284533 5468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:28.287959 5468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:28.288180 5468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:28.295873 5468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36975
I20260620 21:59:28.295885 5535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36975 every 8 connection(s)
I20260620 21:59:28.296862 5468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:28.299705 5536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:28.304487 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5468
I20260620 21:59:28.304662 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:28.304937 17679 external_mini_cluster.cc:1428] Setting key 59c79426e6e6778ce20fdc8fce764147
I20260620 21:59:28.305239 5536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: Bootstrap starting.
I20260620 21:59:28.307765 5536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:28.308811 5536 log.cc:824] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:28.310956 5536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: No bootstrap required, opened a new log
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:59:28.315538 5536 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } }
I20260620 21:59:28.315799 5536 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:28.315879 5536 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0896aa443c34e1da1bdb1439ca9c28f, State: Initialized, Role: FOLLOWER
I20260620 21:59:28.316406 5536 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } }
I20260620 21:59:28.316581 5536 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:28.316660 5536 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:28.316759 5536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:28.317978 5536 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } }
I20260620 21:59:28.318354 5536 leader_election.cc:302] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0896aa443c34e1da1bdb1439ca9c28f; no voters:
I20260620 21:59:28.318742 5536 leader_election.cc:288] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:28.318854 5541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:28.319058 5541 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [term 1 LEADER]: Becoming Leader. State: Replica: b0896aa443c34e1da1bdb1439ca9c28f, State: Running, Role: LEADER
I20260620 21:59:28.319418 5541 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } }
I20260620 21:59:28.320120 5536 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:28.321125 5542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } } }
I20260620 21:59:28.321249 5542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:28.321348 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:28.306333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50613 (local address 127.0.0.1:36975)
0620 21:59:28.306734 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:28.306753 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:28.306796 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:28.307405 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:28.307421 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:28.307765 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:28.308097 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:28.309066 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.310242 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.311430 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.311929 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.312112 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.315131 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:28.315136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:28.315149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:28.315176 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:28.317319 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:28.317894 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:28.317902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:28.317911 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:28.318062 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:28.318562 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:28.318565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:28.318572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:28.319139 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:28.319433 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:28.320056 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 21:59:28.320289 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260620 21:59:28.321496 5542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0896aa443c34e1da1bdb1439ca9c28f. Latest consensus state: current_term: 1 leader_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0896aa443c34e1da1bdb1439ca9c28f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36975 } } }
I20260620 21:59:28.322490 5542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:28.324572 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 21:59:28.325037 5556 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:28.325121 5556 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:28.325186 5550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:28.326140 5550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:28.331828 5550 catalog_manager.cc:1347] Generated new cluster ID: 22cab386fd144f0690bcd19907bc354b
I20260620 21:59:28.331892 5550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:28.335886 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:28.326830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50639 (local address 127.0.0.1:36975)
0620 21:59:28.326924 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:28.326928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:28.327166 (+ 238us) server_negotiation.cc:407] Connection header received
0620 21:59:28.327277 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:28.327279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:28.327327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:28.327411 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:28.328221 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.328986 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.330220 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:28.330523 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:28.330642 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:28.332178 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:28.332182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:28.332185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:28.332208 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:28.333896 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:28.334413 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:28.334417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:28.334419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:28.334508 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:28.334835 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:28.334839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:28.334843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:28.335051 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:28.335200 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:28.335460 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 21:59:28.335732 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 21:59:28.352504 5550 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:28.353726 5550 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:28.362430 5550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0896aa443c34e1da1bdb1439ca9c28f: Generated new TSK 0
I20260620 21:59:28.363224 5550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:28.372826 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5468
W20260620 21:59:28.377648 5453 connection.cc:535] client connection to 127.0.0.1:36975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36975): Connection reset by peer (error 104)
2026-06-20T21:59:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5567](info): set up 2 sockets
Jun 20 21:59:28 dist-test-slave-cx8d krb5kdc[5567](info): commencing operation
krb5kdc: starting...
W20260620 21:59:29.394057 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:29 dist-test-slave-cx8d krb5kdc[5567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:29.521158 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:29.631322 5580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:29.631654 5580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:29.631731 5580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:29.635787 5580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:29.635890 5580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:29.635931 5580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:29.635988 5580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:29.636029 5580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:29.641263 5580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:29.642374 5580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:29.643414 5580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:29.652004 5586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:29.651997 5588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:29.651975 5585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:29.652252 5580 server_base.cc:1034] running on GCE node
I20260620 21:59:29.652980 5580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:29.654186 5580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:29.655381 5580 hybrid_clock.cc:648] HybridClock initialized: now 1781992769655369 us; error 34 us; skew 500 ppm
Jun 20 21:59:29 dist-test-slave-cx8d krb5kdc[5567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:29.658429 5580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:29.659334 5580 webserver.cc:460] Webserver started at http://127.0.0.1:39075/ using document root <none> and password file <none>
I20260620 21:59:29.659934 5580 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:29.660033 5580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:29.660266 5580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:29.662385 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "f86c2c44e1d94a5e9c592cd634d16ed9"
format_stamp: "Formatted at 2026-06-20 21:59:29 on dist-test-slave-cx8d"
server_key: "42c482faa33cae02062911912702c8da"
server_key_iv: "31506a76f9ec049fb81561c3e7fb10ae"
server_key_version: "encryptionkey@0"
I20260620 21:59:29.662985 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "f86c2c44e1d94a5e9c592cd634d16ed9"
format_stamp: "Formatted at 2026-06-20 21:59:29 on dist-test-slave-cx8d"
server_key: "42c482faa33cae02062911912702c8da"
server_key_iv: "31506a76f9ec049fb81561c3e7fb10ae"
server_key_version: "encryptionkey@0"
I20260620 21:59:29.667210 5580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:59:29.669893 5595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:29.671201 5580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:59:29.671368 5580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "f86c2c44e1d94a5e9c592cd634d16ed9"
format_stamp: "Formatted at 2026-06-20 21:59:29 on dist-test-slave-cx8d"
server_key: "42c482faa33cae02062911912702c8da"
server_key_iv: "31506a76f9ec049fb81561c3e7fb10ae"
server_key_version: "encryptionkey@0"
I20260620 21:59:29.671497 5580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:29.694465 5580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:29.698109 5580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:29.698328 5580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:29.705464 5580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41505
I20260620 21:59:29.705536 5647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41505 every 8 connection(s)
I20260620 21:59:29.706522 5580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:29.707947 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5580
I20260620 21:59:29.708115 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:29.708448 17679 external_mini_cluster.cc:1428] Setting key 68eea8d0891684282c033bbb0d28e2f0
I20260620 21:59:29.709797 5648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:29.716773 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9: Bootstrap starting.
I20260620 21:59:29.719009 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:29.709992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35451 (local address 127.0.0.1:41505)
0620 21:59:29.710438 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:29.710468 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:29.710511 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:29.711375 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:29.711398 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:29.711840 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:29.712213 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:29.712637 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.714157 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.715084 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.715816 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.715977 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.716950 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:29.716957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:29.716973 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:29.717475 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:29.717541 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:29.717603 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:29.717732 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:29.717927 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 21:59:29.718078 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260620 21:59:29.719498 5648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:29.720242 5648 log.cc:824] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:29.722124 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:29.722429 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9: No bootstrap required, opened a new log
I20260620 21:59:29.726855 5648 raft_consensus.cc:348] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260620 21:59:29.727155 5648 raft_consensus.cc:374] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:29.727231 5648 raft_consensus.cc:729] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86c2c44e1d94a5e9c592cd634d16ed9, State: Initialized, Role: FOLLOWER
I20260620 21:59:29.727735 5648 consensus_queue.cc:260] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260620 21:59:29.727916 5648 raft_consensus.cc:388] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:29.727999 5648 raft_consensus.cc:482] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:29.728116 5648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:29.728549 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:29.724130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46771 (local address 127.0.0.1:41505)
0620 21:59:29.724241 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:29.724244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:29.724527 (+ 283us) server_negotiation.cc:407] Connection header received
0620 21:59:29.724659 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:29.724661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:29.724698 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:29.724761 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:29.725066 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.725556 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.726724 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:29.726958 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:29.727042 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:29.727988 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:29.727992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:29.727995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:29.728013 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:29.728058 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:29.728073 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:29.728154 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:29.728349 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 21:59:29.728383 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:59:29.729238 5648 raft_consensus.cc:504] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260620 21:59:29.729655 5648 leader_election.cc:302] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86c2c44e1d94a5e9c592cd634d16ed9; no voters:
I20260620 21:59:29.730000 5648 leader_election.cc:288] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:29.730118 5658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:29.730325 5658 raft_consensus.cc:686] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [term 1 LEADER]: Becoming Leader. State: Replica: f86c2c44e1d94a5e9c592cd634d16ed9, State: Running, Role: LEADER
I20260620 21:59:29.730675 5658 consensus_queue.cc:237] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } }
I20260620 21:59:29.731253 5648 sys_catalog.cc:564] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:29.732340 5660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f86c2c44e1d94a5e9c592cd634d16ed9. Latest consensus state: current_term: 1 leader_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } } }
I20260620 21:59:29.732474 5660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:29.732873 5667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:29.732823 5659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86c2c44e1d94a5e9c592cd634d16ed9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41505 } } }
I20260620 21:59:29.732997 5659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:29.736235 5667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:29.742915 5667 catalog_manager.cc:1347] Generated new cluster ID: 7d2c6a1420284ea4ae33d0dc9ce64e10
I20260620 21:59:29.743036 5667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:29.778261 5667 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:29.779254 5667 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:29.784046 5667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f86c2c44e1d94a5e9c592cd634d16ed9: Generated new TSK 0
I20260620 21:59:29.784770 5667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:29.795290 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5580
W20260620 21:59:29.800311 5565 connection.cc:535] client connection to 127.0.0.1:41505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41505): Connection reset by peer (error 104)
2026-06-20T21:59:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:29 dist-test-slave-cx8d krb5kdc[5679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:29 dist-test-slave-cx8d krb5kdc[5679](info): set up 2 sockets
Jun 20 21:59:29 dist-test-slave-cx8d krb5kdc[5679](info): commencing operation
krb5kdc: starting...
W20260620 21:59:30.897920 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:30 dist-test-slave-cx8d krb5kdc[5679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:31.024971 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:31.133249 5692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:31.133533 5692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:31.133656 5692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:31.137621 5692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:31.137682 5692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:31.137702 5692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:31.137718 5692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:31.137733 5692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:31.143038 5692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:31.144061 5692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:31.145025 5692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:31.153682 5700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:31.153687 5697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:31.153687 5698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:31.154140 5692 server_base.cc:1034] running on GCE node
I20260620 21:59:31.154713 5692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:31.155901 5692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:31.157107 5692 hybrid_clock.cc:648] HybridClock initialized: now 1781992771157097 us; error 31 us; skew 500 ppm
Jun 20 21:59:31 dist-test-slave-cx8d krb5kdc[5679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:31.160059 5692 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:31.160944 5692 webserver.cc:460] Webserver started at http://127.0.0.1:44075/ using document root <none> and password file <none>
I20260620 21:59:31.161486 5692 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:31.161547 5692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:31.161800 5692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:31.163826 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "70d526c8df8649adb8c77097c5d3d440"
format_stamp: "Formatted at 2026-06-20 21:59:31 on dist-test-slave-cx8d"
server_key: "0fe0ab512744da3226d97d9a2609395c"
server_key_iv: "e22eac1af6fc1d6bedbfa5d88607ece8"
server_key_version: "encryptionkey@0"
I20260620 21:59:31.164331 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "70d526c8df8649adb8c77097c5d3d440"
format_stamp: "Formatted at 2026-06-20 21:59:31 on dist-test-slave-cx8d"
server_key: "0fe0ab512744da3226d97d9a2609395c"
server_key_iv: "e22eac1af6fc1d6bedbfa5d88607ece8"
server_key_version: "encryptionkey@0"
I20260620 21:59:31.168090 5692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:59:31.170545 5707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:31.171948 5692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:31.172060 5692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "70d526c8df8649adb8c77097c5d3d440"
format_stamp: "Formatted at 2026-06-20 21:59:31 on dist-test-slave-cx8d"
server_key: "0fe0ab512744da3226d97d9a2609395c"
server_key_iv: "e22eac1af6fc1d6bedbfa5d88607ece8"
server_key_version: "encryptionkey@0"
I20260620 21:59:31.172147 5692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:31.176703 5692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:31.179920 5692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:31.180068 5692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:31.187121 5692 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40445
I20260620 21:59:31.187142 5759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40445 every 8 connection(s)
I20260620 21:59:31.188090 5692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:31.191015 5760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:31.191251 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5692
I20260620 21:59:31.191395 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:31.191732 17679 external_mini_cluster.cc:1428] Setting key 25ca817b0d6ef0180cf357b00c231376
I20260620 21:59:31.197184 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:31.193219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54873 (local address 127.0.0.1:40445)
0620 21:59:31.193802 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:31.193817 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:31.193860 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:31.194543 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:31.194560 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:31.194841 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:31.195195 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:31.196005 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:31.196010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:31.196024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:31.196034 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:31.196086 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:31.196105 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:31.196263 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:31.196332 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 21:59:31.196492 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":149,"threads_started":1}
I20260620 21:59:31.198072 5760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: Bootstrap starting.
I20260620 21:59:31.200043 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:31.200546 5760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:31.201299 5760 log.cc:824] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:31.203787 5760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: No bootstrap required, opened a new log
I20260620 21:59:31.203794 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:31.202237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43815 (local address 127.0.0.1:40445)
0620 21:59:31.202360 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:31.202364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:31.202502 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:59:31.202576 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:31.202579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:31.202610 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:31.202695 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:31.203382 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:31.203385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:31.203388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:31.203399 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:31.203427 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:31.203442 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:31.203546 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:31.203652 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 21:59:31.203687 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:59:31.207767 5760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260620 21:59:31.208026 5760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:31.208065 5760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d526c8df8649adb8c77097c5d3d440, State: Initialized, Role: FOLLOWER
I20260620 21:59:31.208514 5760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260620 21:59:31.208647 5760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:31.208690 5760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:31.208765 5760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:31.209978 5760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260620 21:59:31.210348 5760 leader_election.cc:302] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d526c8df8649adb8c77097c5d3d440; no voters:
I20260620 21:59:31.210685 5760 leader_election.cc:288] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:31.210851 5770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:31.211091 5770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [term 1 LEADER]: Becoming Leader. State: Replica: 70d526c8df8649adb8c77097c5d3d440, State: Running, Role: LEADER
I20260620 21:59:31.211483 5770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } }
I20260620 21:59:31.211957 5760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:31.213200 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d526c8df8649adb8c77097c5d3d440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } } }
I20260620 21:59:31.213321 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:31.213759 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d526c8df8649adb8c77097c5d3d440. Latest consensus state: current_term: 1 leader_uuid: "70d526c8df8649adb8c77097c5d3d440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d526c8df8649adb8c77097c5d3d440" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40445 } } }
I20260620 21:59:31.213915 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:31.216580 5785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:31.216673 5785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:31.216758 5779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:31.217414 5779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:31.223218 5779 catalog_manager.cc:1347] Generated new cluster ID: 543da68f8f4643979adea87c9f492eb3
I20260620 21:59:31.223294 5779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:31.239562 5779 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:31.239912 5779 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:31.254356 5779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d526c8df8649adb8c77097c5d3d440: Generated new TSK 0
I20260620 21:59:31.255254 5779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:31.262568 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5692
2026-06-20T21:59:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:31 dist-test-slave-cx8d krb5kdc[5791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:31 dist-test-slave-cx8d krb5kdc[5791](info): set up 2 sockets
Jun 20 21:59:31 dist-test-slave-cx8d krb5kdc[5791](info): commencing operation
krb5kdc: starting...
W20260620 21:59:32.325829 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:32.450320 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:32.563601 5804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:32.563887 5804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:32.563935 5804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:32.568447 5804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:32.568516 5804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:32.568537 5804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:32.568557 5804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:32.568573 5804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:32.574275 5804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:32.575340 5804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:32.576349 5804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:59:32.585453 5804 server_base.cc:1034] running on GCE node
W20260620 21:59:32.585368 5812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:32.585373 5809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:32.585358 5810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:32.586400 5804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:32.587549 5804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:32.588728 5804 hybrid_clock.cc:648] HybridClock initialized: now 1781992772588710 us; error 34 us; skew 500 ppm
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:32.595495 5804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:32.596560 5804 webserver.cc:460] Webserver started at http://127.0.0.1:39109/ using document root <none> and password file <none>
I20260620 21:59:32.597157 5804 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:32.597221 5804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:32.597419 5804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:32.599717 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "51663347887c427a87db0723a75f363b"
format_stamp: "Formatted at 2026-06-20 21:59:32 on dist-test-slave-cx8d"
server_key: "10bdb6ebd99051949b9d3abf40d0abb4"
server_key_iv: "881c1744aca16cf63389b53822343ca4"
server_key_version: "encryptionkey@0"
I20260620 21:59:32.600251 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "51663347887c427a87db0723a75f363b"
format_stamp: "Formatted at 2026-06-20 21:59:32 on dist-test-slave-cx8d"
server_key: "10bdb6ebd99051949b9d3abf40d0abb4"
server_key_iv: "881c1744aca16cf63389b53822343ca4"
server_key_version: "encryptionkey@0"
I20260620 21:59:32.604269 5804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:59:32.607053 5819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:32.608397 5804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:59:32.608582 5804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "51663347887c427a87db0723a75f363b"
format_stamp: "Formatted at 2026-06-20 21:59:32 on dist-test-slave-cx8d"
server_key: "10bdb6ebd99051949b9d3abf40d0abb4"
server_key_iv: "881c1744aca16cf63389b53822343ca4"
server_key_version: "encryptionkey@0"
I20260620 21:59:32.608692 5804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:32.652011 5804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:32.655474 5804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:32.655714 5804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:32.663309 5804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37483
I20260620 21:59:32.663318 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37483 every 8 connection(s)
I20260620 21:59:32.664301 5804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:32.667227 5872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:32.667233 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5804
I20260620 21:59:32.667434 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:32.667763 17679 external_mini_cluster.cc:1428] Setting key 3a979cc1f3ba7bbeb1b710956afa819e
I20260620 21:59:32.674839 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b: Bootstrap starting.
I20260620 21:59:32.677416 5872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b: Neither blocks nor log segments found. Creating new log.
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:59:32.678321 5872 log.cc:824] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:32.680797 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b: No bootstrap required, opened a new log
I20260620 21:59:32.683466 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:32.669226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41075 (local address 127.0.0.1:37483)
0620 21:59:32.669726 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:32.669743 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:32.669795 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:59:32.670488 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:32.670512 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:32.670994 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:32.671374 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:32.672080 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.673627 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.674654 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.675324 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.675482 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.678205 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:32.678211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:32.678229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:32.678262 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:32.680423 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:32.680945 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:32.680956 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:32.680962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:32.681080 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:32.681428 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:32.681433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:32.681435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:32.681920 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:32.682137 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:32.682433 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:59:32.682732 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260620 21:59:32.684808 5872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260620 21:59:32.685096 5872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:32.685148 5872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51663347887c427a87db0723a75f363b, State: Initialized, Role: FOLLOWER
I20260620 21:59:32.685644 5872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260620 21:59:32.685818 5872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:32.685906 5872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:32.686017 5872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:32.686678 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:32.687192 5872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260620 21:59:32.687544 5872 leader_election.cc:302] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51663347887c427a87db0723a75f363b; no voters:
I20260620 21:59:32.687853 5872 leader_election.cc:288] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:32.688009 5877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:32.688293 5877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [term 1 LEADER]: Becoming Leader. State: Replica: 51663347887c427a87db0723a75f363b, State: Running, Role: LEADER
I20260620 21:59:32.688651 5877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } }
I20260620 21:59:32.689263 5872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:32.689990 5877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51663347887c427a87db0723a75f363b. Latest consensus state: current_term: 1 leader_uuid: "51663347887c427a87db0723a75f363b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260620 21:59:32.690121 5877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:32.690651 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51663347887c427a87db0723a75f363b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51663347887c427a87db0723a75f363b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37483 } } }
I20260620 21:59:32.690848 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:32.691154 5891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:32.694531 5891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:32.698917 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:32.689027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59919 (local address 127.0.0.1:37483)
0620 21:59:32.689136 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:32.689140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:32.689432 (+ 292us) server_negotiation.cc:407] Connection header received
0620 21:59:32.689646 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:32.689649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:32.689711 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:32.689800 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:32.690763 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.691608 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.692690 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:32.693045 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:32.693147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:32.694945 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:32.694949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:32.694952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:32.694977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:32.696817 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:32.697284 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:32.697289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:32.697291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:32.697391 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:32.697720 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:32.697723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:32.697725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:32.698134 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:32.698309 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:32.698578 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 21:59:32.698700 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260620 21:59:32.701529 5891 catalog_manager.cc:1347] Generated new cluster ID: 6014d7990286412abdbc8c77a263007f
I20260620 21:59:32.701735 5891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:32.721927 5891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:32.722947 5891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:32.729508 5891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51663347887c427a87db0723a75f363b: Generated new TSK 0
I20260620 21:59:32.730307 5891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:32.734354 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5804
2026-06-20T21:59:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5903](info): set up 2 sockets
Jun 20 21:59:32 dist-test-slave-cx8d krb5kdc[5903](info): commencing operation
krb5kdc: starting...
W20260620 21:59:33.785801 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:33 dist-test-slave-cx8d krb5kdc[5903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:33.920842 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:34.033391 5916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:34.033718 5916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:34.033795 5916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:34.037810 5916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:34.037915 5916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:34.037955 5916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:34.037994 5916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:34.038029 5916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:34.043794 5916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:34.044979 5916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:34.046085 5916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:34.054720 5922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:34.054718 5921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:34.054718 5924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:34.055162 5916 server_base.cc:1034] running on GCE node
I20260620 21:59:34.055733 5916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:34.056949 5916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:34.058133 5916 hybrid_clock.cc:648] HybridClock initialized: now 1781992774058101 us; error 44 us; skew 500 ppm
Jun 20 21:59:34 dist-test-slave-cx8d krb5kdc[5903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:34.061177 5916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:34.062135 5916 webserver.cc:460] Webserver started at http://127.0.0.1:43135/ using document root <none> and password file <none>
I20260620 21:59:34.062680 5916 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:34.062744 5916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:34.062917 5916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:34.065059 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "861a8ecce317498a96695bb5197ccea0"
format_stamp: "Formatted at 2026-06-20 21:59:34 on dist-test-slave-cx8d"
server_key: "bce541c308f14b55a79899c341bbad64"
server_key_iv: "c280a8d6578dafe5514cd21dd9aeb456"
server_key_version: "encryptionkey@0"
I20260620 21:59:34.065685 5916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "861a8ecce317498a96695bb5197ccea0"
format_stamp: "Formatted at 2026-06-20 21:59:34 on dist-test-slave-cx8d"
server_key: "bce541c308f14b55a79899c341bbad64"
server_key_iv: "c280a8d6578dafe5514cd21dd9aeb456"
server_key_version: "encryptionkey@0"
I20260620 21:59:34.069700 5916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:59:34.072252 5931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:34.073771 5916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:34.073921 5916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "861a8ecce317498a96695bb5197ccea0"
format_stamp: "Formatted at 2026-06-20 21:59:34 on dist-test-slave-cx8d"
server_key: "bce541c308f14b55a79899c341bbad64"
server_key_iv: "c280a8d6578dafe5514cd21dd9aeb456"
server_key_version: "encryptionkey@0"
I20260620 21:59:34.074059 5916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:34.101550 5916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:34.105278 5916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:34.105526 5916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:34.113461 5916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34315
I20260620 21:59:34.113529 5983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34315 every 8 connection(s)
I20260620 21:59:34.114519 5916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:34.117246 5984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:34.117739 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 5916
I20260620 21:59:34.117904 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:34.118232 17679 external_mini_cluster.cc:1428] Setting key 96cf6be922db617f8db2b3e96b91874e
I20260620 21:59:34.123373 5984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: Bootstrap starting.
I20260620 21:59:34.126034 5984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:34.126879 5984 log.cc:824] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:34.127696 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:34.119740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36787 (local address 127.0.0.1:34315)
0620 21:59:34.120170 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:34.120197 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:34.120234 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:59:34.120901 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:34.120929 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:34.121289 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:34.121644 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:34.121993 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.123050 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.124255 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.124767 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.124895 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.125783 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:34.125789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:34.125802 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:34.126296 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:34.126355 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:34.126379 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:34.126467 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:34.126625 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 21:59:34.126878 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
I20260620 21:59:34.128911 5984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: No bootstrap required, opened a new log
I20260620 21:59:34.130760 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:34.132926 5984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260620 21:59:34.133220 5984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:34.133333 5984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861a8ecce317498a96695bb5197ccea0, State: Initialized, Role: FOLLOWER
I20260620 21:59:34.133852 5984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260620 21:59:34.134019 5984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:34.134086 5984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:34.134179 5984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:34.135095 5984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260620 21:59:34.135434 5984 leader_election.cc:302] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861a8ecce317498a96695bb5197ccea0; no voters:
I20260620 21:59:34.135771 5984 leader_election.cc:288] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:34.135838 5994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:34.136044 5994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [term 1 LEADER]: Becoming Leader. State: Replica: 861a8ecce317498a96695bb5197ccea0, State: Running, Role: LEADER
I20260620 21:59:34.136374 5994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260620 21:59:34.137063 5984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:34.137598 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:34.132840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36781 (local address 127.0.0.1:34315)
0620 21:59:34.132981 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:34.132986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:34.133180 (+ 194us) server_negotiation.cc:407] Connection header received
0620 21:59:34.133317 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:34.133321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:34.133370 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:34.133479 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:34.133796 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.134534 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.135507 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:34.135877 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:34.135989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:34.137049 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:34.137053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:34.137056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:34.137069 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:34.137104 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:34.137115 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:34.137208 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:34.137365 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:59:34.137429 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:59:34.137537 5994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861a8ecce317498a96695bb5197ccea0. Latest consensus state: current_term: 1 leader_uuid: "861a8ecce317498a96695bb5197ccea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260620 21:59:34.138139 5995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861a8ecce317498a96695bb5197ccea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861a8ecce317498a96695bb5197ccea0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260620 21:59:34.138283 5994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:34.138366 5995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:34.141976 6009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:34.142050 6009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:34.142136 6003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:34.142905 6003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:34.148617 6003 catalog_manager.cc:1347] Generated new cluster ID: a0a8e78632f54994b59a9d96ac25d69d
I20260620 21:59:34.148722 6003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:34.171766 6003 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:34.172755 6003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:34.180133 6003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861a8ecce317498a96695bb5197ccea0: Generated new TSK 0
I20260620 21:59:34.180935 6003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:34.193091 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 5916
2026-06-20T21:59:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:34 dist-test-slave-cx8d krb5kdc[6015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:34 dist-test-slave-cx8d krb5kdc[6015](info): set up 2 sockets
Jun 20 21:59:34 dist-test-slave-cx8d krb5kdc[6015](info): commencing operation
krb5kdc: starting...
W20260620 21:59:35.302206 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:35 dist-test-slave-cx8d krb5kdc[6015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:35.433403 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:35.545554 6028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:35.545933 6028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:35.546021 6028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:35.550129 6028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:35.550246 6028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:35.550285 6028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:35.550321 6028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:35.550354 6028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:35.555740 6028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:35.556908 6028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:35.558014 6028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:35.566843 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:35.566865 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:35.566875 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:35.567020 6028 server_base.cc:1034] running on GCE node
I20260620 21:59:35.567727 6028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:35.568912 6028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:35.570106 6028 hybrid_clock.cc:648] HybridClock initialized: now 1781992775570081 us; error 40 us; skew 500 ppm
Jun 20 21:59:35 dist-test-slave-cx8d krb5kdc[6015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:35.572890 6028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:35.573954 6028 webserver.cc:460] Webserver started at http://127.0.0.1:43741/ using document root <none> and password file <none>
I20260620 21:59:35.574533 6028 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:35.574630 6028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:35.574877 6028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:35.576917 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "9258b9ee4eb8483383fa411ea027749e"
format_stamp: "Formatted at 2026-06-20 21:59:35 on dist-test-slave-cx8d"
server_key: "e07710cd189ccb47faf0384e1cc4216c"
server_key_iv: "de8f5773ae43647dcffa11a2d95a0ad7"
server_key_version: "encryptionkey@0"
I20260620 21:59:35.577507 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "9258b9ee4eb8483383fa411ea027749e"
format_stamp: "Formatted at 2026-06-20 21:59:35 on dist-test-slave-cx8d"
server_key: "e07710cd189ccb47faf0384e1cc4216c"
server_key_iv: "de8f5773ae43647dcffa11a2d95a0ad7"
server_key_version: "encryptionkey@0"
I20260620 21:59:35.581282 6028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:35.583858 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:35.585100 6028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:35.585253 6028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "9258b9ee4eb8483383fa411ea027749e"
format_stamp: "Formatted at 2026-06-20 21:59:35 on dist-test-slave-cx8d"
server_key: "e07710cd189ccb47faf0384e1cc4216c"
server_key_iv: "de8f5773ae43647dcffa11a2d95a0ad7"
server_key_version: "encryptionkey@0"
I20260620 21:59:35.585403 6028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:35.590063 6028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:35.593266 6028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:35.593475 6028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:35.600795 6028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40707
I20260620 21:59:35.600821 6095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40707 every 8 connection(s)
I20260620 21:59:35.601756 6028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:35.604524 6096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:35.610013 6096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e: Bootstrap starting.
I20260620 21:59:35.610169 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6028
I20260620 21:59:35.610318 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:35.610646 17679 external_mini_cluster.cc:1428] Setting key ca5d3ae732b6e16dd0da126436ee0b46
I20260620 21:59:35.612579 6096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:35.613312 6096 log.cc:824] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:35.615572 6096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e: No bootstrap required, opened a new log
I20260620 21:59:35.615779 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:35.611979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38151 (local address 127.0.0.1:40707)
0620 21:59:35.612443 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:35.612457 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:35.612492 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:35.613182 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:35.613199 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:35.613520 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:35.613896 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:35.614602 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:35.614611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:35.614622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:35.614632 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:35.614681 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:35.614700 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:35.614817 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:35.614902 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 21:59:35.615051 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260620 21:59:35.618441 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:35.619403 6096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260620 21:59:35.619704 6096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:35.619776 6096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9258b9ee4eb8483383fa411ea027749e, State: Initialized, Role: FOLLOWER
I20260620 21:59:35.620190 6096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260620 21:59:35.620340 6096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:35.620429 6096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:35.620529 6096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:35.621812 6096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260620 21:59:35.622206 6096 leader_election.cc:302] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9258b9ee4eb8483383fa411ea027749e; no voters:
I20260620 21:59:35.622404 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:35.620512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34477 (local address 127.0.0.1:40707)
0620 21:59:35.620602 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:35.620606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:35.620994 (+ 388us) server_negotiation.cc:407] Connection header received
0620 21:59:35.621093 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:35.621095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:35.621126 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:35.621193 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:35.621978 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:35.621981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:35.621984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:35.621995 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:35.622025 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:35.622035 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:35.622105 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:35.622222 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 21:59:35.622253 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 21:59:35.622593 6096 leader_election.cc:288] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:35.622670 6106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:35.622848 6106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [term 1 LEADER]: Becoming Leader. State: Replica: 9258b9ee4eb8483383fa411ea027749e, State: Running, Role: LEADER
I20260620 21:59:35.623150 6106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260620 21:59:35.623828 6096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:35.624812 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9258b9ee4eb8483383fa411ea027749e. Latest consensus state: current_term: 1 leader_uuid: "9258b9ee4eb8483383fa411ea027749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } } }
I20260620 21:59:35.624948 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:35.625205 6107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9258b9ee4eb8483383fa411ea027749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9258b9ee4eb8483383fa411ea027749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } } }
I20260620 21:59:35.625280 6107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:35.625334 6115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:35.628479 6115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:35.634440 6115 catalog_manager.cc:1347] Generated new cluster ID: 5c8f5b5ec5224184b357d48d2b189f66
I20260620 21:59:35.634541 6115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:35.656405 6115 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:35.656788 6115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:35.662995 6115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9258b9ee4eb8483383fa411ea027749e: Generated new TSK 0
I20260620 21:59:35.663739 6115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:35.670503 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6028
2026-06-20T21:59:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:35 dist-test-slave-cx8d krb5kdc[6127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:35 dist-test-slave-cx8d krb5kdc[6127](info): set up 2 sockets
Jun 20 21:59:35 dist-test-slave-cx8d krb5kdc[6127](info): commencing operation
krb5kdc: starting...
W20260620 21:59:36.706218 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:36 dist-test-slave-cx8d krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:36.832871 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:36.941406 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:36.941784 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:36.941874 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:36.945856 6140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:36.945948 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:36.945983 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:36.946020 6140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:36.946051 6140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:36.951342 6140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:36.952410 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:36.953445 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:36.961931 6148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:36.961916 6146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:36.961946 6145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:36.962158 6140 server_base.cc:1034] running on GCE node
I20260620 21:59:36.962826 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:36.963929 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:36.965137 6140 hybrid_clock.cc:648] HybridClock initialized: now 1781992776965109 us; error 47 us; skew 500 ppm
Jun 20 21:59:36 dist-test-slave-cx8d krb5kdc[6127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:36.968331 6140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:36.969334 6140 webserver.cc:460] Webserver started at http://127.0.0.1:45145/ using document root <none> and password file <none>
I20260620 21:59:36.970007 6140 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:36.970106 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:36.970356 6140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:36.972553 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "00d0972ada7c43a5b3c7571e95db8bc9"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-cx8d"
server_key: "fa1ed1f55eafd5912ef0cc3e9a77c02e"
server_key_iv: "aff11599caeb1371aaeeb4acfbf7ee17"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.973174 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "00d0972ada7c43a5b3c7571e95db8bc9"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-cx8d"
server_key: "fa1ed1f55eafd5912ef0cc3e9a77c02e"
server_key_iv: "aff11599caeb1371aaeeb4acfbf7ee17"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.977098 6140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:59:36.980223 6155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:36.981722 6140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:36.981880 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "00d0972ada7c43a5b3c7571e95db8bc9"
format_stamp: "Formatted at 2026-06-20 21:59:36 on dist-test-slave-cx8d"
server_key: "fa1ed1f55eafd5912ef0cc3e9a77c02e"
server_key_iv: "aff11599caeb1371aaeeb4acfbf7ee17"
server_key_version: "encryptionkey@0"
I20260620 21:59:36.982002 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:36.993840 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:36.997097 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:36.997336 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:37.005949 6140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34999
I20260620 21:59:37.005980 6207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34999 every 8 connection(s)
I20260620 21:59:37.007027 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:37.009261 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6140
I20260620 21:59:37.009403 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:37.009754 17679 external_mini_cluster.cc:1428] Setting key d034fbdf7485ffbb04dae614b05dea04
I20260620 21:59:37.010284 6208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:37.017397 6208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9: Bootstrap starting.
Jun 20 21:59:37 dist-test-slave-cx8d krb5kdc[6127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:59:37.019703 6208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:37.020639 6208 log.cc:824] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:37.023185 6208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9: No bootstrap required, opened a new log
I20260620 21:59:37.025823 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.011228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:57407 (local address 127.0.0.1:34999)
0620 21:59:37.011689 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.011705 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.011741 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:59:37.012395 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.012411 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.012757 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.013037 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.014005 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.015013 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.016278 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.016981 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.017147 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.020082 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.020088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.020105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:37.020137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.022299 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:37.022845 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:37.022852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:37.022859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:37.022996 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:37.023493 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:37.023496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:37.023497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:37.023990 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.024198 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.024523 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.024747 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":151,"threads_started":1}
I20260620 21:59:37.027612 6208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260620 21:59:37.027889 6208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:37.027968 6208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d0972ada7c43a5b3c7571e95db8bc9, State: Initialized, Role: FOLLOWER
I20260620 21:59:37.028478 6208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260620 21:59:37.028649 6208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:37.028719 6208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:37.028822 6208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:37.029064 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:37.029887 6208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260620 21:59:37.030234 6208 leader_election.cc:302] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d0972ada7c43a5b3c7571e95db8bc9; no voters:
I20260620 21:59:37.030606 6208 leader_election.cc:288] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:37.030676 6217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:37.030869 6217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 00d0972ada7c43a5b3c7571e95db8bc9, State: Running, Role: LEADER
I20260620 21:59:37.031191 6217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } }
I20260620 21:59:37.031935 6208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:37.033084 6218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d0972ada7c43a5b3c7571e95db8bc9. Latest consensus state: current_term: 1 leader_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } } }
I20260620 21:59:37.033241 6218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:37.033536 6219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0972ada7c43a5b3c7571e95db8bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34999 } } }
I20260620 21:59:37.033661 6227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:37.033715 6219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:37.037487 6227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:37.040997 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:37.031228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:44855 (local address 127.0.0.1:34999)
0620 21:59:37.031341 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:37.031345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:37.031883 (+ 538us) server_negotiation.cc:407] Connection header received
0620 21:59:37.032031 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:37.032034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:37.032079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:37.032173 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:37.033079 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.033942 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.035002 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:37.035296 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:37.035435 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:37.037101 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:37.037105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:37.037108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:37.037133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:37.038853 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:37.039415 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:37.039419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:37.039421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:37.039511 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:37.039913 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:37.039916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:37.039918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:37.040145 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:37.040298 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:37.040634 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 21:59:37.040727 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:59:37.043897 6227 catalog_manager.cc:1347] Generated new cluster ID: 21bfa80a15894614b9e16f6b2b463018
I20260620 21:59:37.044013 6227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:37.049917 6227 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:37.050833 6227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:37.059432 6227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d0972ada7c43a5b3c7571e95db8bc9: Generated new TSK 0
I20260620 21:59:37.060186 6227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:37.065440 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6140
W20260620 21:59:37.070732 6125 connection.cc:535] client connection to 127.0.0.1:34999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34999): Connection reset by peer (error 104)
2026-06-20T21:59:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:37 dist-test-slave-cx8d krb5kdc[6239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:37 dist-test-slave-cx8d krb5kdc[6239](info): set up 2 sockets
Jun 20 21:59:37 dist-test-slave-cx8d krb5kdc[6239](info): commencing operation
krb5kdc: starting...
W20260620 21:59:38.129842 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:38 dist-test-slave-cx8d krb5kdc[6239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:38.256524 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:38.369740 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:38.370042 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:38.370118 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:38.374265 6252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:38.374329 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:38.374351 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:38.374367 6252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:38.374430 6252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:38.379871 6252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:38.380983 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:38.382117 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:38.390529 6260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:38.390538 6258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:38.390538 6257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:38.390733 6252 server_base.cc:1034] running on GCE node
I20260620 21:59:38.391389 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:38.392659 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:38.393853 6252 hybrid_clock.cc:648] HybridClock initialized: now 1781992778393842 us; error 28 us; skew 500 ppm
Jun 20 21:59:38 dist-test-slave-cx8d krb5kdc[6239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:38.396719 6252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:38.397697 6252 webserver.cc:460] Webserver started at http://127.0.0.1:41441/ using document root <none> and password file <none>
I20260620 21:59:38.398269 6252 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:38.398404 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:38.398639 6252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:38.400728 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "88bc6d5579aa4e33ae1d470262a9b501"
format_stamp: "Formatted at 2026-06-20 21:59:38 on dist-test-slave-cx8d"
server_key: "f1c724b284d21398900871e1d67e8c9e"
server_key_iv: "ed25f9a42919d987bfbfdf63881a7823"
server_key_version: "encryptionkey@0"
I20260620 21:59:38.401311 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "88bc6d5579aa4e33ae1d470262a9b501"
format_stamp: "Formatted at 2026-06-20 21:59:38 on dist-test-slave-cx8d"
server_key: "f1c724b284d21398900871e1d67e8c9e"
server_key_iv: "ed25f9a42919d987bfbfdf63881a7823"
server_key_version: "encryptionkey@0"
I20260620 21:59:38.405309 6252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:59:38.407972 6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:38.409305 6252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:38.409508 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "88bc6d5579aa4e33ae1d470262a9b501"
format_stamp: "Formatted at 2026-06-20 21:59:38 on dist-test-slave-cx8d"
server_key: "f1c724b284d21398900871e1d67e8c9e"
server_key_iv: "ed25f9a42919d987bfbfdf63881a7823"
server_key_version: "encryptionkey@0"
I20260620 21:59:38.409684 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:38.434008 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:38.437590 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:38.437811 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:38.444806 6252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46381
I20260620 21:59:38.444852 6319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46381 every 8 connection(s)
I20260620 21:59:38.445829 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:38.448776 6320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:38.453342 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6252
I20260620 21:59:38.453527 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:38.453850 17679 external_mini_cluster.cc:1428] Setting key dbed0e98aef839b2ba225bcbfc54a6b4
I20260620 21:59:38.454550 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: Bootstrap starting.
I20260620 21:59:38.458093 6320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:38.458879 6320 log.cc:824] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:38.461174 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: No bootstrap required, opened a new log
I20260620 21:59:38.463572 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:38.455382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:55423 (local address 127.0.0.1:46381)
0620 21:59:38.455861 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:38.455882 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:38.455925 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:38.456618 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:38.456636 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:38.457011 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:38.457312 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:38.457748 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:38.458851 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:38.459958 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:38.460439 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:38.460564 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:38.461496 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:38.461501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:38.461513 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:38.461964 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:38.462021 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:38.462044 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:38.462146 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:38.462455 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 21:59:38.462622 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":110,"threads_started":1}
I20260620 21:59:38.464673 6320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } }
I20260620 21:59:38.464948 6320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:38.465037 6320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bc6d5579aa4e33ae1d470262a9b501, State: Initialized, Role: FOLLOWER
I20260620 21:59:38.465554 6320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } }
I20260620 21:59:38.465752 6320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:38.465860 6320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:38.465981 6320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:38.466921 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:38.467144 6320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } }
I20260620 21:59:38.467494 6320 leader_election.cc:302] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88bc6d5579aa4e33ae1d470262a9b501; no voters:
I20260620 21:59:38.467819 6320 leader_election.cc:288] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:38.467963 6325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:38.468214 6325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [term 1 LEADER]: Becoming Leader. State: Replica: 88bc6d5579aa4e33ae1d470262a9b501, State: Running, Role: LEADER
I20260620 21:59:38.468673 6325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } }
I20260620 21:59:38.469226 6320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:38.470077 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88bc6d5579aa4e33ae1d470262a9b501. Latest consensus state: current_term: 1 leader_uuid: "88bc6d5579aa4e33ae1d470262a9b501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } } }
I20260620 21:59:38.470219 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:38.470932 6326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88bc6d5579aa4e33ae1d470262a9b501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bc6d5579aa4e33ae1d470262a9b501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46381 } } }
I20260620 21:59:38.471024 6326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:38.474948 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:38.469311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:45829 (local address 127.0.0.1:46381)
0620 21:59:38.469420 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:38.469425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:38.469973 (+ 548us) server_negotiation.cc:407] Connection header received
0620 21:59:38.470198 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:38.470202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:38.470260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:38.470321 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:38.470857 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:38.471710 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:38.472747 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:38.473138 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:38.473251 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:38.474265 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:38.474270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:38.474273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:38.474288 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:38.474355 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:38.474372 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:38.474581 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:38.474768 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 21:59:38.474807 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 21:59:38.475081 6345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:38.475176 6345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:38.475349 6339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:38.476235 6339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:38.481773 6339 catalog_manager.cc:1347] Generated new cluster ID: 56134c1918b146a7b424eb195d1a0b03
I20260620 21:59:38.481851 6339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:38.490120 6339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:38.491119 6339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:38.501063 6339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88bc6d5579aa4e33ae1d470262a9b501: Generated new TSK 0
I20260620 21:59:38.501904 6339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:38.510881 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6252
W20260620 21:59:38.516064 6237 connection.cc:535] client connection to 127.0.0.1:46381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46381): Connection reset by peer (error 104)
2026-06-20T21:59:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:38 dist-test-slave-cx8d krb5kdc[6351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:38 dist-test-slave-cx8d krb5kdc[6351](info): set up 2 sockets
Jun 20 21:59:38 dist-test-slave-cx8d krb5kdc[6351](info): commencing operation
krb5kdc: starting...
W20260620 21:59:39.554339 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:39 dist-test-slave-cx8d krb5kdc[6351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:39.681133 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:39.794530 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:39.794842 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:39.794926 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:39.798977 6364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:39.799060 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:39.799109 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:39.799150 6364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:39.799204 6364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:39.804472 6364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:39.805554 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:39.806653 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:39.815060 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:39.815109 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.815164 6364 server_base.cc:1034] running on GCE node
W20260620 21:59:39.815060 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:39.815865 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:39.816890 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:39.818049 6364 hybrid_clock.cc:648] HybridClock initialized: now 1781992779818016 us; error 47 us; skew 500 ppm
Jun 20 21:59:39 dist-test-slave-cx8d krb5kdc[6351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:39.820765 6364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:39.821769 6364 webserver.cc:460] Webserver started at http://127.0.0.1:39807/ using document root <none> and password file <none>
I20260620 21:59:39.822324 6364 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:39.822415 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:39.822644 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:39.824695 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-cx8d"
server_key: "a8f06580fe7c31fa6b43a6606da78f25"
server_key_iv: "84192deb124c21d92115261986c4577e"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.825250 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-cx8d"
server_key: "a8f06580fe7c31fa6b43a6606da78f25"
server_key_iv: "84192deb124c21d92115261986c4577e"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.828862 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:59:39.831435 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:39.832906 6364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:59:39.833063 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a"
format_stamp: "Formatted at 2026-06-20 21:59:39 on dist-test-slave-cx8d"
server_key: "a8f06580fe7c31fa6b43a6606da78f25"
server_key_iv: "84192deb124c21d92115261986c4577e"
server_key_version: "encryptionkey@0"
I20260620 21:59:39.833181 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:39.837913 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:39.841259 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:39.841491 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:39.848855 6364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38173
I20260620 21:59:39.848881 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38173 every 8 connection(s)
I20260620 21:59:39.849870 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:39.852617 6432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:39.857633 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6364
I20260620 21:59:39.857810 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:39.858105 17679 external_mini_cluster.cc:1428] Setting key 82da4faad4561bd041698c4a478da50f
I20260620 21:59:39.858367 6432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: Bootstrap starting.
I20260620 21:59:39.861042 6432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:39.861949 6432 log.cc:824] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:39.863592 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.859562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:45135 (local address 127.0.0.1:38173)
0620 21:59:39.860000 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.860018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.860060 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:59:39.860880 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.860904 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.861288 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.861675 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.862403 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.862411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.862428 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:39.862439 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.862489 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:39.862507 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.862637 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.862763 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.862923 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260620 21:59:39.864035 6432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: No bootstrap required, opened a new log
I20260620 21:59:39.866310 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:39.868458 6432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260620 21:59:39.868794 6432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:39.868863 6432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e95d4d05bb46e7ab7fae5be9cbaa5a, State: Initialized, Role: FOLLOWER
I20260620 21:59:39.869390 6432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260620 21:59:39.869591 6432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:39.870023 6432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:39.870143 6432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:39.870800 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:39.868609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:43537 (local address 127.0.0.1:38173)
0620 21:59:39.868772 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:39.868776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:39.869112 (+ 336us) server_negotiation.cc:407] Connection header received
0620 21:59:39.869154 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:39.869158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:39.869197 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:39.869293 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:39.870152 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:39.870158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:39.870161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:39.870174 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:39.870203 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:39.870217 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:39.870278 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:39.870488 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:59:39.870566 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:59:39.871217 6432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260620 21:59:39.871587 6432 leader_election.cc:302] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24e95d4d05bb46e7ab7fae5be9cbaa5a; no voters:
I20260620 21:59:39.871915 6432 leader_election.cc:288] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:39.872181 6442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:39.872469 6442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [term 1 LEADER]: Becoming Leader. State: Replica: 24e95d4d05bb46e7ab7fae5be9cbaa5a, State: Running, Role: LEADER
I20260620 21:59:39.872872 6442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260620 21:59:39.873222 6432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:39.874518 6444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260620 21:59:39.874645 6444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:39.874883 6444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24e95d4d05bb46e7ab7fae5be9cbaa5a. Latest consensus state: current_term: 1 leader_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e95d4d05bb46e7ab7fae5be9cbaa5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260620 21:59:39.874974 6444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:39.879129 6457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:39.879216 6457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:39.879274 6451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:39.880056 6451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:39.886369 6451 catalog_manager.cc:1347] Generated new cluster ID: 22f0908ad48540e7bc3b1d6c29d97818
I20260620 21:59:39.886451 6451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:39.893039 6451 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:39.893400 6451 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:39.900048 6451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24e95d4d05bb46e7ab7fae5be9cbaa5a: Generated new TSK 0
I20260620 21:59:39.900903 6451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:39.906131 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6364
2026-06-20T21:59:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:39 dist-test-slave-cx8d krb5kdc[6463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:39 dist-test-slave-cx8d krb5kdc[6463](info): set up 2 sockets
Jun 20 21:59:39 dist-test-slave-cx8d krb5kdc[6463](info): commencing operation
krb5kdc: starting...
W20260620 21:59:40.945890 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:41.066927 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:41.176930 6476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:41.177243 6476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:41.177328 6476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:41.181322 6476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:41.181430 6476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:41.181471 6476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:41.181530 6476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:41.181586 6476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:41.186829 6476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:41.187911 6476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:41.188947 6476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:41.197057 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:41.197078 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:41.197083 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:41.197381 6476 server_base.cc:1034] running on GCE node
I20260620 21:59:41.198024 6476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:41.199150 6476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:41.200341 6476 hybrid_clock.cc:648] HybridClock initialized: now 1781992781200324 us; error 29 us; skew 500 ppm
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:41.203047 6476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:41.203979 6476 webserver.cc:460] Webserver started at http://127.0.0.1:40327/ using document root <none> and password file <none>
I20260620 21:59:41.204536 6476 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:41.204628 6476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:41.204896 6476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:41.206979 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "c44dc45e04a24e1994346df98c324612"
format_stamp: "Formatted at 2026-06-20 21:59:41 on dist-test-slave-cx8d"
server_key: "2c33d8cf4b885b3758715edee8521586"
server_key_iv: "342c799ccaf8239f3ee0d250b73c04a4"
server_key_version: "encryptionkey@0"
I20260620 21:59:41.207545 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "c44dc45e04a24e1994346df98c324612"
format_stamp: "Formatted at 2026-06-20 21:59:41 on dist-test-slave-cx8d"
server_key: "2c33d8cf4b885b3758715edee8521586"
server_key_iv: "342c799ccaf8239f3ee0d250b73c04a4"
server_key_version: "encryptionkey@0"
I20260620 21:59:41.211264 6476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:41.213749 6491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:41.214969 6476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:41.215109 6476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "c44dc45e04a24e1994346df98c324612"
format_stamp: "Formatted at 2026-06-20 21:59:41 on dist-test-slave-cx8d"
server_key: "2c33d8cf4b885b3758715edee8521586"
server_key_iv: "342c799ccaf8239f3ee0d250b73c04a4"
server_key_version: "encryptionkey@0"
I20260620 21:59:41.215219 6476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:41.227288 6476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:41.230103 6476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:41.230297 6476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:41.237370 6476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42947
I20260620 21:59:41.237396 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42947 every 8 connection(s)
I20260620 21:59:41.238287 6476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:41.240917 6544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:41.242831 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6476
I20260620 21:59:41.243005 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:41.243317 17679 external_mini_cluster.cc:1428] Setting key 0619f2e561a2711d725b74f4c2783fac
I20260620 21:59:41.247857 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: Bootstrap starting.
I20260620 21:59:41.250483 6544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:41.251291 6544 log.cc:824] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:59:41.253551 6544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: No bootstrap required, opened a new log
I20260620 21:59:41.258277 6544 raft_consensus.cc:348] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } }
I20260620 21:59:41.258530 6544 raft_consensus.cc:374] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:41.258575 6544 raft_consensus.cc:729] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44dc45e04a24e1994346df98c324612, State: Initialized, Role: FOLLOWER
I20260620 21:59:41.258834 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:41.244701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59665 (local address 127.0.0.1:42947)
0620 21:59:41.245116 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:41.245134 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:41.245176 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:59:41.246013 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:41.246040 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:41.246484 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:41.246864 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:41.247564 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:41.249005 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:41.249903 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:41.250570 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:41.250757 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:41.253634 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:41.253640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:41.253657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:41.253687 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:41.255644 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:41.256184 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:41.256189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:41.256195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:41.256341 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:41.256653 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:41.256657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:41.256659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:41.257162 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:41.257387 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:41.257643 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 21:59:41.257882 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260620 21:59:41.259037 6544 consensus_queue.cc:260] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } }
I20260620 21:59:41.259195 6544 raft_consensus.cc:388] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:41.259271 6544 raft_consensus.cc:482] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:41.259369 6544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:41.260361 6544 raft_consensus.cc:504] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } }
I20260620 21:59:41.260697 6544 leader_election.cc:302] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c44dc45e04a24e1994346df98c324612; no voters:
I20260620 21:59:41.261060 6544 leader_election.cc:288] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:41.261179 6549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:41.261392 6549 raft_consensus.cc:686] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [term 1 LEADER]: Becoming Leader. State: Replica: c44dc45e04a24e1994346df98c324612, State: Running, Role: LEADER
I20260620 21:59:41.261737 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } }
I20260620 21:59:41.262010 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:41.262444 6544 sys_catalog.cc:564] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:41.262874 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44dc45e04a24e1994346df98c324612. Latest consensus state: current_term: 1 leader_uuid: "c44dc45e04a24e1994346df98c324612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } } }
I20260620 21:59:41.262995 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:41.263275 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c44dc45e04a24e1994346df98c324612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c44dc45e04a24e1994346df98c324612" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42947 } } }
I20260620 21:59:41.263379 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:41.267489 6568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:41.267572 6568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:41.267686 6555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:41.268690 6555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:41.273648 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:41.264362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:39193 (local address 127.0.0.1:42947)
0620 21:59:41.264463 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:41.264468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:41.265145 (+ 677us) server_negotiation.cc:407] Connection header received
0620 21:59:41.265183 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:41.265185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:41.265246 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:41.265323 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:41.266140 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:41.266652 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:41.267488 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:41.267839 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:41.267939 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:41.269806 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:41.269810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:41.269813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:41.269849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:41.271552 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:41.272121 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:41.272125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:41.272127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:41.272215 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:41.272586 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:41.272589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:41.272590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:41.272804 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:41.272966 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:41.273347 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 21:59:41.273448 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 21:59:41.274770 6555 catalog_manager.cc:1347] Generated new cluster ID: 368aaa4393a94c3fb46bbbe8ad57b083
I20260620 21:59:41.274828 6555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:41.290951 6555 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:41.291890 6555 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:41.316442 6555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c44dc45e04a24e1994346df98c324612: Generated new TSK 0
I20260620 21:59:41.317268 6555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:41.331429 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6476
W20260620 21:59:41.336699 6461 connection.cc:535] client connection to 127.0.0.1:42947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42947): Connection reset by peer (error 104)
2026-06-20T21:59:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6575](info): set up 2 sockets
Jun 20 21:59:41 dist-test-slave-cx8d krb5kdc[6575](info): commencing operation
krb5kdc: starting...
W20260620 21:59:42.365909 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:42 dist-test-slave-cx8d krb5kdc[6575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:42.492831 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:42.603015 6588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:42.603291 6588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:42.603341 6588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:42.607266 6588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:42.607322 6588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:42.607342 6588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:42.607358 6588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:42.607373 6588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:42.612484 6588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:42.613482 6588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:42.614548 6588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:42.623109 6594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:42.623133 6593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:42.623133 6596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:42.623361 6588 server_base.cc:1034] running on GCE node
I20260620 21:59:42.624119 6588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:42.625247 6588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:42.626417 6588 hybrid_clock.cc:648] HybridClock initialized: now 1781992782626379 us; error 49 us; skew 500 ppm
Jun 20 21:59:42 dist-test-slave-cx8d krb5kdc[6575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:42.629295 6588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:42.630292 6588 webserver.cc:460] Webserver started at http://127.0.0.1:36337/ using document root <none> and password file <none>
I20260620 21:59:42.630836 6588 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:42.630944 6588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:42.631172 6588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:42.633188 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "55680210e0ed48ac962c4d48d5be3357"
format_stamp: "Formatted at 2026-06-20 21:59:42 on dist-test-slave-cx8d"
server_key: "eb79a8d37cdc1e3b6c5e6558f4374c51"
server_key_iv: "c128efdc43ef290faba6a3032f6ccfaa"
server_key_version: "encryptionkey@0"
I20260620 21:59:42.633780 6588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "55680210e0ed48ac962c4d48d5be3357"
format_stamp: "Formatted at 2026-06-20 21:59:42 on dist-test-slave-cx8d"
server_key: "eb79a8d37cdc1e3b6c5e6558f4374c51"
server_key_iv: "c128efdc43ef290faba6a3032f6ccfaa"
server_key_version: "encryptionkey@0"
I20260620 21:59:42.637475 6588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:59:42.639887 6603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:42.641145 6588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:42.641284 6588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "55680210e0ed48ac962c4d48d5be3357"
format_stamp: "Formatted at 2026-06-20 21:59:42 on dist-test-slave-cx8d"
server_key: "eb79a8d37cdc1e3b6c5e6558f4374c51"
server_key_iv: "c128efdc43ef290faba6a3032f6ccfaa"
server_key_version: "encryptionkey@0"
I20260620 21:59:42.641424 6588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:42.671082 6588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:42.674224 6588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:42.674386 6588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:42.681711 6588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42475
I20260620 21:59:42.681751 6655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42475 every 8 connection(s)
I20260620 21:59:42.682754 6588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:42.685530 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:42.689762 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6588
I20260620 21:59:42.689908 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:42.690200 17679 external_mini_cluster.cc:1428] Setting key c15382f956f6341146744f72de1d667b
I20260620 21:59:42.691463 6656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357: Bootstrap starting.
I20260620 21:59:42.693751 6656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:42.694459 6656 log.cc:824] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:42.696228 6656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357: No bootstrap required, opened a new log
I20260620 21:59:42.699235 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:42.691683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:49993 (local address 127.0.0.1:42475)
0620 21:59:42.692128 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:42.692142 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:42.692177 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:42.692861 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:42.692878 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:42.693218 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:42.693607 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:42.694019 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:42.695027 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:42.695923 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:42.696445 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:42.696602 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:42.697557 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:42.697604 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:42.697622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:42.698017 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:42.698072 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:42.698096 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:42.698240 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:42.698423 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 21:59:42.698561 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260620 21:59:42.700002 6656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 21:59:42.700234 6656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:42.700287 6656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55680210e0ed48ac962c4d48d5be3357, State: Initialized, Role: FOLLOWER
I20260620 21:59:42.700707 6656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 21:59:42.700863 6656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:42.700942 6656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:42.701059 6656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:42.702065 6656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 21:59:42.702183 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:42.702409 6656 leader_election.cc:302] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55680210e0ed48ac962c4d48d5be3357; no voters:
I20260620 21:59:42.702739 6656 leader_election.cc:288] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:42.702801 6661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:42.702986 6661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [term 1 LEADER]: Becoming Leader. State: Replica: 55680210e0ed48ac962c4d48d5be3357, State: Running, Role: LEADER
I20260620 21:59:42.703274 6661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } }
I20260620 21:59:42.703965 6656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:42.704928 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55680210e0ed48ac962c4d48d5be3357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } } }
I20260620 21:59:42.705070 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:42.705386 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55680210e0ed48ac962c4d48d5be3357. Latest consensus state: current_term: 1 leader_uuid: "55680210e0ed48ac962c4d48d5be3357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55680210e0ed48ac962c4d48d5be3357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42475 } } }
I20260620 21:59:42.705475 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:42.705760 6675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:42.708986 6675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:42.709904 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:42.704631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:49779 (local address 127.0.0.1:42475)
0620 21:59:42.704773 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:42.704778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:42.705030 (+ 252us) server_negotiation.cc:407] Connection header received
0620 21:59:42.705299 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:42.705302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:42.705348 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:42.705431 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:42.705894 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:42.706744 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:42.707824 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:42.708167 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:42.708292 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:42.709228 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:42.709232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:42.709235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:42.709247 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:42.709275 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:42.709289 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:42.709374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:42.709734 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 21:59:42.709779 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:42.714422 6675 catalog_manager.cc:1347] Generated new cluster ID: c8d570d2df8c43a388203070f228a033
I20260620 21:59:42.714498 6675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:42.727149 6675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:42.728107 6675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:42.742492 6675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55680210e0ed48ac962c4d48d5be3357: Generated new TSK 0
I20260620 21:59:42.743261 6675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:42.748179 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6588
W20260620 21:59:42.753772 6573 connection.cc:535] client connection to 127.0.0.1:42475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42475): Connection reset by peer (error 104)
2026-06-20T21:59:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:42 dist-test-slave-cx8d krb5kdc[6687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:42 dist-test-slave-cx8d krb5kdc[6687](info): set up 2 sockets
Jun 20 21:59:42 dist-test-slave-cx8d krb5kdc[6687](info): commencing operation
krb5kdc: starting...
W20260620 21:59:43.790360 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:43 dist-test-slave-cx8d krb5kdc[6687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:59:43.918550 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:44.028587 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:44.028870 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:44.028913 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:44.033926 6700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:44.034034 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:44.034071 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:44.034113 6700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:44.034148 6700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:44.040534 6700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:44.041803 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:44.042876 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:44.051694 6706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:44.051708 6705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:44.051728 6708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:44.051961 6700 server_base.cc:1034] running on GCE node
I20260620 21:59:44.052922 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:44.054112 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:44.055342 6700 hybrid_clock.cc:648] HybridClock initialized: now 1781992784055301 us; error 47 us; skew 500 ppm
Jun 20 21:59:44 dist-test-slave-cx8d krb5kdc[6687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:44.063082 6700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:59:44.064116 6700 webserver.cc:460] Webserver started at http://127.0.0.1:43035/ using document root <none> and password file <none>
I20260620 21:59:44.064692 6700 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:44.064797 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:44.065023 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:44.067091 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "a6e8db630f4b455a9693dce4c77bc7d9"
format_stamp: "Formatted at 2026-06-20 21:59:44 on dist-test-slave-cx8d"
server_key: "b498f8f8dc272e4e3297cfd3ffd0fda0"
server_key_iv: "83e60359b79ead96a64561e5bfc3d05a"
server_key_version: "encryptionkey@0"
I20260620 21:59:44.067664 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "a6e8db630f4b455a9693dce4c77bc7d9"
format_stamp: "Formatted at 2026-06-20 21:59:44 on dist-test-slave-cx8d"
server_key: "b498f8f8dc272e4e3297cfd3ffd0fda0"
server_key_iv: "83e60359b79ead96a64561e5bfc3d05a"
server_key_version: "encryptionkey@0"
I20260620 21:59:44.071511 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:59:44.074102 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:44.075492 6700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:44.075652 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "a6e8db630f4b455a9693dce4c77bc7d9"
format_stamp: "Formatted at 2026-06-20 21:59:44 on dist-test-slave-cx8d"
server_key: "b498f8f8dc272e4e3297cfd3ffd0fda0"
server_key_iv: "83e60359b79ead96a64561e5bfc3d05a"
server_key_version: "encryptionkey@0"
I20260620 21:59:44.075781 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:44.080950 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:44.084095 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:44.084306 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:44.091805 6700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46735
I20260620 21:59:44.091820 6767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46735 every 8 connection(s)
I20260620 21:59:44.092770 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:44.095685 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:44.096132 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6700
I20260620 21:59:44.096277 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:44.096585 17679 external_mini_cluster.cc:1428] Setting key 9eb2d2d2f60d046418bde5f9d5fad78a
I20260620 21:59:44.102135 6768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9: Bootstrap starting.
I20260620 21:59:44.102154 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:44.098196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:41463 (local address 127.0.0.1:46735)
0620 21:59:44.098678 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:44.098693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:44.098731 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:59:44.099416 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:44.099433 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:44.099729 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:44.100106 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:44.100898 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:44.100903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:44.100915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:44.100925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:44.100979 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:44.100998 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:44.101147 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:44.101227 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 21:59:44.101392 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":119,"threads_started":1}
I20260620 21:59:44.104457 6768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:44.104981 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:44.105289 6768 log.cc:824] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:44.107316 6768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9: No bootstrap required, opened a new log
I20260620 21:59:44.109151 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:44.107300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:32933 (local address 127.0.0.1:46735)
0620 21:59:44.107439 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:44.107445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:44.107683 (+ 238us) server_negotiation.cc:407] Connection header received
0620 21:59:44.107830 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:44.107834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:44.107867 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:44.107938 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:44.108736 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:44.108739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:44.108741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:44.108753 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:44.108778 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:44.108789 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:44.108865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:44.108962 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:59:44.108996 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:59:44.111514 6768 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260620 21:59:44.111776 6768 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:44.111855 6768 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e8db630f4b455a9693dce4c77bc7d9, State: Initialized, Role: FOLLOWER
I20260620 21:59:44.112274 6768 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260620 21:59:44.112424 6768 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:44.112511 6768 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:44.112608 6768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:44.113523 6768 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260620 21:59:44.113894 6768 leader_election.cc:302] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e8db630f4b455a9693dce4c77bc7d9; no voters:
I20260620 21:59:44.114229 6768 leader_election.cc:288] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:44.114316 6778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:44.114499 6778 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [term 1 LEADER]: Becoming Leader. State: Replica: a6e8db630f4b455a9693dce4c77bc7d9, State: Running, Role: LEADER
I20260620 21:59:44.114787 6778 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260620 21:59:44.115584 6768 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:44.116569 6780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e8db630f4b455a9693dce4c77bc7d9. Latest consensus state: current_term: 1 leader_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260620 21:59:44.116705 6780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:44.117060 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e8db630f4b455a9693dce4c77bc7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260620 21:59:44.117123 6787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:44.117223 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:44.120538 6787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:44.125931 6787 catalog_manager.cc:1347] Generated new cluster ID: adc061f473df472aa1105668b8c2298e
I20260620 21:59:44.126009 6787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:44.137739 6787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:44.138108 6787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:44.149163 6787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e8db630f4b455a9693dce4c77bc7d9: Generated new TSK 0
I20260620 21:59:44.149993 6787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:44.155251 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6700
2026-06-20T21:59:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:44 dist-test-slave-cx8d krb5kdc[6799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:44 dist-test-slave-cx8d krb5kdc[6799](info): set up 2 sockets
Jun 20 21:59:44 dist-test-slave-cx8d krb5kdc[6799](info): commencing operation
krb5kdc: starting...
W20260620 21:59:45.250309 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:45.370843 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41179
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:45.478714 6812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:45.478986 6812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:45.479027 6812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:45.483047 6812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:45.483106 6812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:45.483125 6812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:45.483141 6812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:45.483157 6812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:45.488394 6812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:45.489476 6812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:45.490525 6812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:45.498953 6820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:45.498961 6817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:45.498965 6818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:45.499215 6812 server_base.cc:1034] running on GCE node
I20260620 21:59:45.499742 6812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:45.500919 6812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:45.502112 6812 hybrid_clock.cc:648] HybridClock initialized: now 1781992785502087 us; error 49 us; skew 500 ppm
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992785, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:45.504904 6812 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:45.505837 6812 webserver.cc:460] Webserver started at http://127.17.67.254:33741/ using document root <none> and password file <none>
I20260620 21:59:45.506356 6812 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:45.506414 6812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:45.506578 6812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:45.508515 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "ef542ee1f71d408995f987e8949281b3"
format_stamp: "Formatted at 2026-06-20 21:59:45 on dist-test-slave-cx8d"
server_key: "f86af0e9e2dba0d9c67f1070fb43b0a3"
server_key_iv: "7e55805e08307d3f11215170fce62cc2"
server_key_version: "encryptionkey@0"
I20260620 21:59:45.509019 6812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "ef542ee1f71d408995f987e8949281b3"
format_stamp: "Formatted at 2026-06-20 21:59:45 on dist-test-slave-cx8d"
server_key: "f86af0e9e2dba0d9c67f1070fb43b0a3"
server_key_iv: "7e55805e08307d3f11215170fce62cc2"
server_key_version: "encryptionkey@0"
I20260620 21:59:45.512610 6812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:59:45.515177 6827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:45.516571 6812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:59:45.516726 6812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "ef542ee1f71d408995f987e8949281b3"
format_stamp: "Formatted at 2026-06-20 21:59:45 on dist-test-slave-cx8d"
server_key: "f86af0e9e2dba0d9c67f1070fb43b0a3"
server_key_iv: "7e55805e08307d3f11215170fce62cc2"
server_key_version: "encryptionkey@0"
I20260620 21:59:45.516844 6812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:45.530802 6812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:45.534132 6812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:45.534349 6812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:45.541819 6812 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41179
I20260620 21:59:45.541850 6879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41179 every 8 connection(s)
I20260620 21:59:45.542809 6812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:45.545535 6880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:45.546844 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6812
I20260620 21:59:45.546993 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:45.547286 17679 external_mini_cluster.cc:1428] Setting key d240dac3c8f18af3ec553a5ad1699a89
I20260620 21:59:45.551666 6880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: Bootstrap starting.
I20260620 21:59:45.553870 6880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:45.554584 6880 log.cc:824] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:45.556471 6880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: No bootstrap required, opened a new log
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:45.560323 6880 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } }
I20260620 21:59:45.560604 6880 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:45.560667 6880 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef542ee1f71d408995f987e8949281b3, State: Initialized, Role: FOLLOWER
I20260620 21:59:45.561110 6880 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } }
I20260620 21:59:45.561231 6880 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:45.561326 6880 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:45.561467 6880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:45.562541 6880 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } }
I20260620 21:59:45.562830 6880 leader_election.cc:302] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef542ee1f71d408995f987e8949281b3; no voters:
I20260620 21:59:45.563133 6880 leader_election.cc:288] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:45.563202 6885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:45.563246 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:45.548840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51923 (local address 127.17.67.254:41179)
0620 21:59:45.549281 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:45.549298 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:45.549341 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:45.550100 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:45.550120 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:45.550480 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:45.550864 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:45.551797 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:45.552899 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:45.553919 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:45.554484 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:45.554673 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:45.557560 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:45.557608 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:45.557637 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:45.557666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:45.559634 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:45.560241 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:45.560247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:45.560251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:45.560416 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:45.560810 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:45.560814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:45.560816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:45.561418 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:45.561692 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:45.562014 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 21:59:45.562297 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260620 21:59:45.563398 6885 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [term 1 LEADER]: Becoming Leader. State: Replica: ef542ee1f71d408995f987e8949281b3, State: Running, Role: LEADER
I20260620 21:59:45.563690 6885 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } }
I20260620 21:59:45.564379 6880 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:45.565389 6887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef542ee1f71d408995f987e8949281b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } } }
I20260620 21:59:45.565836 6887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:45.565698 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef542ee1f71d408995f987e8949281b3. Latest consensus state: current_term: 1 leader_uuid: "ef542ee1f71d408995f987e8949281b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef542ee1f71d408995f987e8949281b3" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41179 } } }
I20260620 21:59:45.566174 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:45.566311 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 21:59:45.569267 6900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:45.569358 6900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:45.569432 6895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:45.570175 6895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:45.576403 6895 catalog_manager.cc:1347] Generated new cluster ID: 89fe59aa1547464881d692e4e7a82309
I20260620 21:59:45.576462 6895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:45.577919 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:45.568727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54053 (local address 127.17.67.254:41179)
0620 21:59:45.568838 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:45.568843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:45.568861 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:59:45.568897 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:45.568901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:45.568942 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:45.569023 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:45.570758 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:45.571547 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:45.572414 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:45.572644 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:45.572748 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:45.574205 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:45.574209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:45.574212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:45.574242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:45.575823 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:45.576283 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:45.576287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:45.576289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:45.576372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:45.576746 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:45.576751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:45.576753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:45.576983 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:45.577139 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:45.577437 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:59:45.577745 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:45.583866 6895 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:45.584918 6895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:45.593688 6895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef542ee1f71d408995f987e8949281b3: Generated new TSK 0
I20260620 21:59:45.594483 6895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:45.599848 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6812
W20260620 21:59:45.605151 6797 connection.cc:535] client connection to 127.17.67.254:41179 recv error: Network error: failed to read from TLS socket (remote: 127.17.67.254:41179): Connection reset by peer (error 104)
2026-06-20T21:59:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6911](info): set up 2 sockets
Jun 20 21:59:45 dist-test-slave-cx8d krb5kdc[6911](info): commencing operation
krb5kdc: starting...
W20260620 21:59:46.737998 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:46 dist-test-slave-cx8d krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:46.868513 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46027
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:46.983817 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:46.984119 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:46.984163 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:46.989218 6924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:46.989306 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:46.989329 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:46.989351 6924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:46.989408 6924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:46.995795 6924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:46027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:46.997047 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:46.998216 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:47.007862 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:47.007867 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:47.007867 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:47.008409 6924 server_base.cc:1034] running on GCE node
I20260620 21:59:47.009042 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:47.010402 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:47.011611 6924 hybrid_clock.cc:648] HybridClock initialized: now 1781992787011576 us; error 49 us; skew 500 ppm
Jun 20 21:59:47 dist-test-slave-cx8d krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992787, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:47.014905 6924 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:47.016018 6924 webserver.cc:460] Webserver started at http://127.17.67.254:33837/ using document root <none> and password file <none>
I20260620 21:59:47.016649 6924 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:47.016752 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:47.017016 6924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:47.019381 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "d3df854464594ba68ebd6186c39fc316"
format_stamp: "Formatted at 2026-06-20 21:59:47 on dist-test-slave-cx8d"
server_key: "47187721eb7c835b3ec247a69522521a"
server_key_iv: "f2a106124a9986e3c1f28605bd41f51b"
server_key_version: "encryptionkey@0"
I20260620 21:59:47.020007 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "d3df854464594ba68ebd6186c39fc316"
format_stamp: "Formatted at 2026-06-20 21:59:47 on dist-test-slave-cx8d"
server_key: "47187721eb7c835b3ec247a69522521a"
server_key_iv: "f2a106124a9986e3c1f28605bd41f51b"
server_key_version: "encryptionkey@0"
I20260620 21:59:47.024324 6924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:59:47.027213 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:47.028697 6924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:59:47.028884 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "d3df854464594ba68ebd6186c39fc316"
format_stamp: "Formatted at 2026-06-20 21:59:47 on dist-test-slave-cx8d"
server_key: "47187721eb7c835b3ec247a69522521a"
server_key_iv: "f2a106124a9986e3c1f28605bd41f51b"
server_key_version: "encryptionkey@0"
I20260620 21:59:47.029027 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:47.064831 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:47.068812 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:47.069056 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:47.077333 6924 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:46027
I20260620 21:59:47.077343 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:46027 every 8 connection(s)
I20260620 21:59:47.078482 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:47.081415 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:47.086283 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 6924
I20260620 21:59:47.086449 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:47.086753 17679 external_mini_cluster.cc:1428] Setting key 6d325d0bc156a97114e86d8cbf087830
I20260620 21:59:47.087415 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316: Bootstrap starting.
I20260620 21:59:47.089953 6992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:47.090703 6992 log.cc:824] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:47.092588 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316: No bootstrap required, opened a new log
I20260620 21:59:47.095808 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:47.088150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35867 (local address 127.17.67.254:46027)
0620 21:59:47.088610 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:47.088624 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:47.088659 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:59:47.089296 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:47.089314 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:47.089751 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:47.090087 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:47.090515 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:47.091542 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:47.092455 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:47.092992 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:47.093171 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:47.094136 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:47.094149 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:47.094161 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:47.094552 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:47.094603 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:47.094625 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:47.094764 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:47.094943 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 21:59:47.095085 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260620 21:59:47.096539 6992 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } }
I20260620 21:59:47.096800 6992 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:47.096870 6992 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3df854464594ba68ebd6186c39fc316, State: Initialized, Role: FOLLOWER
I20260620 21:59:47.097335 6992 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } }
I20260620 21:59:47.097500 6992 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:47.097591 6992 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:47.097719 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:47.098804 6992 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } }
I20260620 21:59:47.099030 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:47.099182 6992 leader_election.cc:302] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3df854464594ba68ebd6186c39fc316; no voters:
I20260620 21:59:47.099515 6992 leader_election.cc:288] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:47.099629 6997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:47.099920 6997 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [term 1 LEADER]: Becoming Leader. State: Replica: d3df854464594ba68ebd6186c39fc316, State: Running, Role: LEADER
I20260620 21:59:47.100318 6997 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } }
I20260620 21:59:47.100862 6992 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:47.101622 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3df854464594ba68ebd6186c39fc316. Latest consensus state: current_term: 1 leader_uuid: "d3df854464594ba68ebd6186c39fc316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } } }
I20260620 21:59:47.101747 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:47.102066 6999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3df854464594ba68ebd6186c39fc316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df854464594ba68ebd6186c39fc316" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 46027 } } }
I20260620 21:59:47.102150 6999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:47.102198 7011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:47.105602 7011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:47.107445 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:47.101234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55107 (local address 127.17.67.254:46027)
0620 21:59:47.101338 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:47.101342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:47.102563 (+ 1221us) server_negotiation.cc:407] Connection header received
0620 21:59:47.102757 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:47.102760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:47.102813 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:47.102871 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:47.103904 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:47.104421 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:47.105590 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:47.105905 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:47.106016 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:47.106952 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:47.106958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:47.106961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:47.106978 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:47.107009 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:47.107026 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:47.107107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:47.107262 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 21:59:47.107298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 21:59:47.112545 7011 catalog_manager.cc:1347] Generated new cluster ID: ee450200f1d24ee39d8c812da1353af8
I20260620 21:59:47.112633 7011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:47.130050 7011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:47.131070 7011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:47.139417 7011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3df854464594ba68ebd6186c39fc316: Generated new TSK 0
I20260620 21:59:47.140245 7011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:47.145844 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 6924
W20260620 21:59:47.151108 6909 connection.cc:535] client connection to 127.17.67.254:46027 recv error: Network error: failed to read from TLS socket (remote: 127.17.67.254:46027): Connection reset by peer (error 104)
2026-06-20T21:59:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:47 dist-test-slave-cx8d krb5kdc[7023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:47 dist-test-slave-cx8d krb5kdc[7023](info): set up 2 sockets
Jun 20 21:59:47 dist-test-slave-cx8d krb5kdc[7023](info): commencing operation
krb5kdc: starting...
W20260620 21:59:48.213825 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:48 dist-test-slave-cx8d krb5kdc[7023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:48.340936 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:42009
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:48.456923 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:48.457194 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:48.457237 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:48.461191 7036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:48.461265 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:48.461289 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:48.461308 7036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:48.461323 7036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:48.466769 7036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:42009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:48.467795 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:48.468773 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:48.477455 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:48.477455 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:48.477646 7036 server_base.cc:1034] running on GCE node
W20260620 21:59:48.477430 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:48.478255 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:48.479348 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:48.480553 7036 hybrid_clock.cc:648] HybridClock initialized: now 1781992788480527 us; error 48 us; skew 500 ppm
Jun 20 21:59:48 dist-test-slave-cx8d krb5kdc[7023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992788, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:48.483472 7036 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:48.484356 7036 webserver.cc:460] Webserver started at http://127.17.67.254:44733/ using document root <none> and password file <none>
I20260620 21:59:48.484922 7036 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:48.485016 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:48.485239 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:48.487361 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "515a2933c54543d4b20fbe4344ee9821"
format_stamp: "Formatted at 2026-06-20 21:59:48 on dist-test-slave-cx8d"
server_key: "8d2377c3e2b556d155e995955e43e3b8"
server_key_iv: "70b07de7741c7e6f3a29ce162f7e29e2"
server_key_version: "encryptionkey@0"
I20260620 21:59:48.487941 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "515a2933c54543d4b20fbe4344ee9821"
format_stamp: "Formatted at 2026-06-20 21:59:48 on dist-test-slave-cx8d"
server_key: "8d2377c3e2b556d155e995955e43e3b8"
server_key_iv: "70b07de7741c7e6f3a29ce162f7e29e2"
server_key_version: "encryptionkey@0"
I20260620 21:59:48.491698 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:48.494119 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:48.495460 7036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:48.495621 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "515a2933c54543d4b20fbe4344ee9821"
format_stamp: "Formatted at 2026-06-20 21:59:48 on dist-test-slave-cx8d"
server_key: "8d2377c3e2b556d155e995955e43e3b8"
server_key_iv: "70b07de7741c7e6f3a29ce162f7e29e2"
server_key_version: "encryptionkey@0"
I20260620 21:59:48.495743 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:48.500566 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:48.503793 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:48.503985 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:48.511361 7036 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:42009
I20260620 21:59:48.511379 7103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:42009 every 8 connection(s)
I20260620 21:59:48.512290 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:48.515004 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:48.517428 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7036
I20260620 21:59:48.517601 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:48.517915 17679 external_mini_cluster.cc:1428] Setting key a7095de9c89f7cfb7fc3bfbf7469c992
I20260620 21:59:48.520951 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: Bootstrap starting.
I20260620 21:59:48.522977 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:48.519363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41299 (local address 127.17.67.254:42009)
0620 21:59:48.519767 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:48.519781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:48.519814 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:59:48.520437 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:48.520454 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:48.520727 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:48.521066 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:48.521843 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:48.521848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:48.521860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:48.521870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:48.521922 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:48.521941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:48.522087 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:48.522155 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 21:59:48.522309 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260620 21:59:48.523419 7104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:48.524116 7104 log.cc:824] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:48.525942 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:48.526196 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: No bootstrap required, opened a new log
I20260620 21:59:48.529731 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:48.528095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49101 (local address 127.17.67.254:42009)
0620 21:59:48.528225 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:48.528229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:48.528395 (+ 166us) server_negotiation.cc:407] Connection header received
0620 21:59:48.528489 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:48.528492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:48.528528 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:48.528610 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:48.529344 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:48.529347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:48.529350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:48.529361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:48.529391 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:48.529407 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:48.529493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:48.529581 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 21:59:48.529619 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:59:48.530206 7104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } }
I20260620 21:59:48.530417 7104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:48.530480 7104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515a2933c54543d4b20fbe4344ee9821, State: Initialized, Role: FOLLOWER
I20260620 21:59:48.530922 7104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } }
I20260620 21:59:48.531047 7104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:48.531093 7104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:48.531162 7104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:48.531966 7104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } }
I20260620 21:59:48.532248 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515a2933c54543d4b20fbe4344ee9821; no voters:
I20260620 21:59:48.532490 7104 leader_election.cc:288] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:48.532636 7114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:48.532896 7114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [term 1 LEADER]: Becoming Leader. State: Replica: 515a2933c54543d4b20fbe4344ee9821, State: Running, Role: LEADER
I20260620 21:59:48.533200 7114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } }
I20260620 21:59:48.533691 7104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:48.535377 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515a2933c54543d4b20fbe4344ee9821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } } }
I20260620 21:59:48.535420 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515a2933c54543d4b20fbe4344ee9821. Latest consensus state: current_term: 1 leader_uuid: "515a2933c54543d4b20fbe4344ee9821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515a2933c54543d4b20fbe4344ee9821" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42009 } } }
I20260620 21:59:48.535586 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:48.535591 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:48.538573 7129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:48.538692 7129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:48.538821 7128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:48.539494 7128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:48.545127 7128 catalog_manager.cc:1347] Generated new cluster ID: 6894d5f67c274501af54109105b2a5fc
I20260620 21:59:48.545207 7128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:48.569940 7128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:48.570295 7128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:48.580329 7128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515a2933c54543d4b20fbe4344ee9821: Generated new TSK 0
I20260620 21:59:48.581173 7128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:48.594237 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7036
2026-06-20T21:59:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:48 dist-test-slave-cx8d krb5kdc[7135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:48 dist-test-slave-cx8d krb5kdc[7135](info): set up 2 sockets
Jun 20 21:59:48 dist-test-slave-cx8d krb5kdc[7135](info): commencing operation
krb5kdc: starting...
W20260620 21:59:49.745955 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:49 dist-test-slave-cx8d krb5kdc[7135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:49.866737 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43257
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:43257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:49.980943 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:49.981220 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:49.981261 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:49.985291 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:49.985363 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:49.985384 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:49.985400 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:49.985416 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:49.990562 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:43257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:49.991532 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:49.992472 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:50.001103 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:50.001081 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:50.001116 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:50.001633 7148 server_base.cc:1034] running on GCE node
I20260620 21:59:50.002224 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:50.003588 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:50.004769 7148 hybrid_clock.cc:648] HybridClock initialized: now 1781992790004751 us; error 36 us; skew 500 ppm
Jun 20 21:59:50 dist-test-slave-cx8d krb5kdc[7135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992790, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:50.011265 7148 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:50.012436 7148 webserver.cc:460] Webserver started at http://127.17.67.254:41189/ using document root <none> and password file <none>
I20260620 21:59:50.013072 7148 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:50.013178 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:50.013479 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:50.015862 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "ec2cd20d0c9047ad8a0b69820c78dd84"
format_stamp: "Formatted at 2026-06-20 21:59:50 on dist-test-slave-cx8d"
server_key: "233a3af669703b775f0132819ce1ca69"
server_key_iv: "53a25ec87bede7e39058f7b9c6e2e2dd"
server_key_version: "encryptionkey@0"
I20260620 21:59:50.016523 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "ec2cd20d0c9047ad8a0b69820c78dd84"
format_stamp: "Formatted at 2026-06-20 21:59:50 on dist-test-slave-cx8d"
server_key: "233a3af669703b775f0132819ce1ca69"
server_key_iv: "53a25ec87bede7e39058f7b9c6e2e2dd"
server_key_version: "encryptionkey@0"
I20260620 21:59:50.020659 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 21:59:50.023314 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:50.024725 7148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:59:50.024895 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "ec2cd20d0c9047ad8a0b69820c78dd84"
format_stamp: "Formatted at 2026-06-20 21:59:50 on dist-test-slave-cx8d"
server_key: "233a3af669703b775f0132819ce1ca69"
server_key_iv: "53a25ec87bede7e39058f7b9c6e2e2dd"
server_key_version: "encryptionkey@0"
I20260620 21:59:50.025020 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:50.038128 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:50.041465 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:50.041735 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:50.049599 7148 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:43257
I20260620 21:59:50.049634 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:43257 every 8 connection(s)
I20260620 21:59:50.050622 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:50.053352 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:50.054179 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7148
I20260620 21:59:50.054328 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:50.054652 17679 external_mini_cluster.cc:1428] Setting key 091010dc435a115d752b18abb6cbe043
I20260620 21:59:50.060238 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84: Bootstrap starting.
I20260620 21:59:50.062566 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:50.063310 7216 log.cc:824] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:59:50 dist-test-slave-cx8d krb5kdc[7135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:50.065377 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84: No bootstrap required, opened a new log
I20260620 21:59:50.068941 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } }
I20260620 21:59:50.069204 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:50.069296 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cd20d0c9047ad8a0b69820c78dd84, State: Initialized, Role: FOLLOWER
I20260620 21:59:50.069798 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } }
I20260620 21:59:50.069954 7216 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:50.069917 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:50.056043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49489 (local address 127.17.67.254:43257)
0620 21:59:50.056579 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:50.056597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:50.056641 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:59:50.057527 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:50.057553 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:50.058074 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:50.058505 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:50.059511 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:50.060558 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:50.061410 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:50.061970 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:50.062144 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:50.064967 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:50.064972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:50.064987 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:50.065022 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:50.066890 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:50.067351 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:50.067361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:50.067366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:50.067470 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:50.067765 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:50.067769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:50.067772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:50.068147 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:50.068369 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:50.068678 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 21:59:50.068968 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":121,"threads_started":1}
I20260620 21:59:50.070072 7216 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:50.070180 7216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:50.071278 7216 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } }
I20260620 21:59:50.071612 7216 leader_election.cc:302] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec2cd20d0c9047ad8a0b69820c78dd84; no voters:
I20260620 21:59:50.071939 7216 leader_election.cc:288] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:50.072048 7221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:50.072309 7221 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [term 1 LEADER]: Becoming Leader. State: Replica: ec2cd20d0c9047ad8a0b69820c78dd84, State: Running, Role: LEADER
I20260620 21:59:50.072651 7221 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } }
I20260620 21:59:50.073036 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:50.073258 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:50.074506 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2cd20d0c9047ad8a0b69820c78dd84. Latest consensus state: current_term: 1 leader_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } } }
I20260620 21:59:50.074633 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:50.074506 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2cd20d0c9047ad8a0b69820c78dd84" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 43257 } } }
I20260620 21:59:50.074990 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:50.075304 7234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:50.078953 7234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:50.083969 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:50.076009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36083 (local address 127.17.67.254:43257)
0620 21:59:50.076132 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:50.076136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:50.076334 (+ 198us) server_negotiation.cc:407] Connection header received
0620 21:59:50.076534 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:50.076537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:50.076588 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:50.076688 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:50.077345 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:50.077936 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:50.078725 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:50.078962 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:50.079076 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:50.080487 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:50.080490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:50.080493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:50.080524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:50.082268 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:50.082762 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:50.082765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:50.082767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:50.082847 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:50.083131 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:50.083136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:50.083138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:50.083316 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:50.083452 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:50.083687 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 21:59:50.083801 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:50.090898 7234 catalog_manager.cc:1347] Generated new cluster ID: 9674c9d90ab441038884dff0e2033de4
I20260620 21:59:50.091029 7234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:50.100167 7234 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:50.101409 7234 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:50.109251 7234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2cd20d0c9047ad8a0b69820c78dd84: Generated new TSK 0
I20260620 21:59:50.110105 7234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:50.120949 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7148
2026-06-20T21:59:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1526 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:50 dist-test-slave-cx8d krb5kdc[7247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:50 dist-test-slave-cx8d krb5kdc[7247](info): set up 2 sockets
Jun 20 21:59:50 dist-test-slave-cx8d krb5kdc[7247](info): commencing operation
krb5kdc: starting...
W20260620 21:59:51.229907 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:51 dist-test-slave-cx8d krb5kdc[7247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:51.356310 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39787
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:51.471427 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:51.471707 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:51.471750 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:51.475857 7260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:51.475919 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:51.475940 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:51.475958 7260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:51.476014 7260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:51.481267 7260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:51.482476 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:51.483525 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:51.491931 7268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:51.492063 7265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:51.491931 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:51.492278 7260 server_base.cc:1034] running on GCE node
I20260620 21:59:51.492861 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:51.494020 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:51.495221 7260 hybrid_clock.cc:648] HybridClock initialized: now 1781992791495204 us; error 29 us; skew 500 ppm
Jun 20 21:59:51 dist-test-slave-cx8d krb5kdc[7247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:51.498030 7260 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:51.498862 7260 webserver.cc:460] Webserver started at http://127.17.67.254:41277/ using document root <none> and password file <none>
I20260620 21:59:51.499388 7260 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:51.499450 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:51.499614 7260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:51.501555 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9"
format_stamp: "Formatted at 2026-06-20 21:59:51 on dist-test-slave-cx8d"
server_key: "2b4daa998446666721e86f8f3e016d2d"
server_key_iv: "55eb78ab525defa632d0dcc78e977613"
server_key_version: "encryptionkey@0"
I20260620 21:59:51.502230 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9"
format_stamp: "Formatted at 2026-06-20 21:59:51 on dist-test-slave-cx8d"
server_key: "2b4daa998446666721e86f8f3e016d2d"
server_key_iv: "55eb78ab525defa632d0dcc78e977613"
server_key_version: "encryptionkey@0"
I20260620 21:59:51.506001 7260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:59:51.508774 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:51.510377 7260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:51.510514 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9"
format_stamp: "Formatted at 2026-06-20 21:59:51 on dist-test-slave-cx8d"
server_key: "2b4daa998446666721e86f8f3e016d2d"
server_key_iv: "55eb78ab525defa632d0dcc78e977613"
server_key_version: "encryptionkey@0"
I20260620 21:59:51.510627 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:51.546630 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:51.550148 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:51.550403 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:51.558171 7260 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39787
I20260620 21:59:51.558190 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39787 every 8 connection(s)
I20260620 21:59:51.559216 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:51.562083 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:51.563160 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7260
I20260620 21:59:51.563362 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:51.563699 17679 external_mini_cluster.cc:1428] Setting key 016780b3ae6c4c4d0bc245a5142b4707
I20260620 21:59:51.567993 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: Bootstrap starting.
I20260620 21:59:51.570225 7328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:51.570984 7328 log.cc:824] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:51.573236 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: No bootstrap required, opened a new log
I20260620 21:59:51.573596 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:51.565215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37407 (local address 127.17.67.254:39787)
0620 21:59:51.565732 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:51.565753 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:51.565791 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:59:51.566483 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:51.566500 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:51.566867 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:51.567219 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:51.567663 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:51.568763 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:51.569832 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:51.570383 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:51.570579 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:51.571554 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:51.571558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:51.571570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:51.572068 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:51.572134 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:51.572163 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:51.572265 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:51.572461 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 21:59:51.572658 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":121,"threads_started":1}
I20260620 21:59:51.576411 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:51.577306 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } }
I20260620 21:59:51.577543 7328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:51.577621 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fffbe73ae34ebaaf8d20b5ab730bf9, State: Initialized, Role: FOLLOWER
I20260620 21:59:51.578068 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } }
I20260620 21:59:51.578217 7328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:51.578320 7328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:51.578439 7328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:51.579619 7328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } }
I20260620 21:59:51.579968 7328 leader_election.cc:302] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17fffbe73ae34ebaaf8d20b5ab730bf9; no voters:
I20260620 21:59:51.580291 7328 leader_election.cc:288] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:51.580387 7338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:51.580621 7338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 17fffbe73ae34ebaaf8d20b5ab730bf9, State: Running, Role: LEADER
I20260620 21:59:51.581010 7338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } }
I20260620 21:59:51.581555 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:51.582856 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } } }
I20260620 21:59:51.583045 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:51.583616 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:51.578496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36923 (local address 127.17.67.254:39787)
0620 21:59:51.578609 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:51.578615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:51.578898 (+ 283us) server_negotiation.cc:407] Connection header received
0620 21:59:51.578986 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:51.578991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:51.579035 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:51.579119 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:51.579362 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:51.580136 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:51.581368 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:51.581747 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:51.581860 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:51.582823 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:51.582829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:51.582834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:51.582849 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:51.582881 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:51.582895 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:51.582969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:51.583426 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 21:59:51.583485 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:51.582366 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17fffbe73ae34ebaaf8d20b5ab730bf9. Latest consensus state: current_term: 1 leader_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fffbe73ae34ebaaf8d20b5ab730bf9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39787 } } }
I20260620 21:59:51.583787 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9 [sys.catalog]: This master's current role is: LEADER
W20260620 21:59:51.586771 7354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:51.586869 7354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:51.586949 7347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:51.587703 7347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:51.593840 7347 catalog_manager.cc:1347] Generated new cluster ID: d14668fd9fd3456a9e8bebcaa39c149e
I20260620 21:59:51.593912 7347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:51.606644 7347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:51.607654 7347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:51.613125 7347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17fffbe73ae34ebaaf8d20b5ab730bf9: Generated new TSK 0
I20260620 21:59:51.613911 7347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:51.624565 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7260
2026-06-20T21:59:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:51 dist-test-slave-cx8d krb5kdc[7359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:51 dist-test-slave-cx8d krb5kdc[7359](info): set up 2 sockets
Jun 20 21:59:51 dist-test-slave-cx8d krb5kdc[7359](info): commencing operation
krb5kdc: starting...
W20260620 21:59:52.705935 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:52 dist-test-slave-cx8d krb5kdc[7359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:52.827997 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:42281
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:41401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:52.943965 7372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:52.944244 7372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:52.944288 7372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:52.948388 7372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:52.948451 7372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:52.948470 7372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:52.948487 7372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:52.948503 7372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:52.953792 7372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:41401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:42281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:52.954821 7372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:52.955909 7372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:52.964653 7378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:52.964640 7380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:52.964636 7377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:52.964957 7372 server_base.cc:1034] running on GCE node
I20260620 21:59:52.965639 7372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:52.966830 7372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:52.968022 7372 hybrid_clock.cc:648] HybridClock initialized: now 1781992792967986 us; error 53 us; skew 500 ppm
Jun 20 21:59:52 dist-test-slave-cx8d krb5kdc[7359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992792, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:52.971176 7372 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:52.972112 7372 webserver.cc:460] Webserver started at http://127.17.67.254:46157/ using document root <none> and password file <none>
I20260620 21:59:52.972675 7372 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:52.972766 7372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:52.973008 7372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:52.975085 7372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "2fb47386ddf34617a16e8cc3cc8028e9"
format_stamp: "Formatted at 2026-06-20 21:59:52 on dist-test-slave-cx8d"
server_key: "f7f6156121fe24a51c78003e0ff6e2a3"
server_key_iv: "87dd606ee468560a10a2b3ec3403d649"
server_key_version: "encryptionkey@0"
I20260620 21:59:52.975633 7372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "2fb47386ddf34617a16e8cc3cc8028e9"
format_stamp: "Formatted at 2026-06-20 21:59:52 on dist-test-slave-cx8d"
server_key: "f7f6156121fe24a51c78003e0ff6e2a3"
server_key_iv: "87dd606ee468560a10a2b3ec3403d649"
server_key_version: "encryptionkey@0"
I20260620 21:59:52.979364 7372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:59:52.981868 7387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:52.983211 7372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:52.983392 7372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "2fb47386ddf34617a16e8cc3cc8028e9"
format_stamp: "Formatted at 2026-06-20 21:59:52 on dist-test-slave-cx8d"
server_key: "f7f6156121fe24a51c78003e0ff6e2a3"
server_key_iv: "87dd606ee468560a10a2b3ec3403d649"
server_key_version: "encryptionkey@0"
I20260620 21:59:52.983518 7372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:52.988598 7372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:52.992266 7372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:52.992484 7372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:53.000109 7372 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:42281
I20260620 21:59:53.000134 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:42281 every 8 connection(s)
I20260620 21:59:53.001082 7372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:53.003798 7440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:53.004403 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7372
I20260620 21:59:53.004546 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:53.004880 17679 external_mini_cluster.cc:1428] Setting key dddc3f4b0bd40e8f36522a1425dcc889
I20260620 21:59:53.010169 7440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9: Bootstrap starting.
I20260620 21:59:53.010336 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:53.006306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43765 (local address 127.17.67.254:42281)
0620 21:59:53.006724 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:53.006742 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:53.006778 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:59:53.007478 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:53.007494 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:53.007787 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:53.008155 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:53.008982 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:53.008987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:53.009000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:53.009011 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:53.009066 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:53.009086 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:53.009246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:53.009324 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:59:53.009488 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260620 21:59:53.012702 7440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:53.013547 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:53.013681 7440 log.cc:824] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:53.016006 7440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9: No bootstrap required, opened a new log
I20260620 21:59:53.018038 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:53.016058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60671 (local address 127.17.67.254:42281)
0620 21:59:53.016234 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:53.016237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:53.016454 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:59:53.016591 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:53.016593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:53.016632 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:53.016701 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:53.017466 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:53.017468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:53.017470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:59:53.017480 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:53.017506 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:59:53.017516 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:53.017683 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:53.017814 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 21:59:53.017886 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:59:53.020884 7440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } }
I20260620 21:59:53.021134 7440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:59:53.021214 7440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb47386ddf34617a16e8cc3cc8028e9, State: Initialized, Role: FOLLOWER
I20260620 21:59:53.021778 7440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } }
I20260620 21:59:53.021935 7440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:59:53.022011 7440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:59:53.022147 7440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:53.023180 7440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } }
I20260620 21:59:53.023556 7440 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb47386ddf34617a16e8cc3cc8028e9; no voters:
I20260620 21:59:53.023890 7440 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:59:53.024024 7450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:53.024297 7450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb47386ddf34617a16e8cc3cc8028e9, State: Running, Role: LEADER
I20260620 21:59:53.024689 7450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } }
I20260620 21:59:53.025370 7440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:53.026556 7451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } } }
I20260620 21:59:53.026693 7451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:53.027102 7459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:53.027089 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb47386ddf34617a16e8cc3cc8028e9. Latest consensus state: current_term: 1 leader_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb47386ddf34617a16e8cc3cc8028e9" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 42281 } } }
I20260620 21:59:53.027812 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:53.030443 7459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:53.036692 7459 catalog_manager.cc:1347] Generated new cluster ID: 4b945964130f41deb0a1e27220947f85
I20260620 21:59:53.036790 7459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:53.052037 7459 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:53.052433 7459 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:53.060861 7459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb47386ddf34617a16e8cc3cc8028e9: Generated new TSK 0
I20260620 21:59:53.061764 7459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:59:53.069954 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7372
2026-06-20T21:59:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1446 ms)
[----------] 18 tests from AuthTokenIssuingTest (26117 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:53 dist-test-slave-cx8d krb5kdc[7471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:53 dist-test-slave-cx8d krb5kdc[7471](info): set up 2 sockets
Jun 20 21:59:53 dist-test-slave-cx8d krb5kdc[7471](info): commencing operation
krb5kdc: starting...
W20260620 21:59:54.142501 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:54.273133 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41609
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:54.393368 7484 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:54.393806 7484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:54.393879 7484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:54.393957 7484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:54.398172 7484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:54.398293 7484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:54.398331 7484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:54.398370 7484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:54.398403 7484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:54.404034 7484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:54.405223 7484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:54.406316 7484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:54.415894 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:54.415932 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.415969 7484 server_base.cc:1034] running on GCE node
W20260620 21:59:54.415894 7492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.416764 7484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:54.417971 7484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:54.419194 7484 hybrid_clock.cc:648] HybridClock initialized: now 1781992794419172 us; error 38 us; skew 500 ppm
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:54.422371 7484 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:54.423326 7484 webserver.cc:460] Webserver started at http://127.17.67.254:42979/ using document root <none> and password file <none>
I20260620 21:59:54.423905 7484 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:54.424013 7484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:54.424252 7484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:54.426452 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "d02a457552f04bc18269f31b9076e46a"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "59da24d6624b8ce51a2fa376cec44178"
server_key_iv: "a67d24de53a24c45d364eff4fa1ff782"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.427027 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "d02a457552f04bc18269f31b9076e46a"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "59da24d6624b8ce51a2fa376cec44178"
server_key_iv: "a67d24de53a24c45d364eff4fa1ff782"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.430907 7484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:59:54.433971 7499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:54.435454 7484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:59:54.435622 7484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "d02a457552f04bc18269f31b9076e46a"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "59da24d6624b8ce51a2fa376cec44178"
server_key_iv: "a67d24de53a24c45d364eff4fa1ff782"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.435760 7484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:54.448396 7484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:54.451898 7484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:54.452113 7484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:54.459465 7484 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:41609
I20260620 21:59:54.459810 7551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:41609 every 8 connection(s)
I20260620 21:59:54.460502 7484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:54.463379 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:54.467228 7552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.470559 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7484
I20260620 21:59:54.470701 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:54.470999 17679 external_mini_cluster.cc:1428] Setting key 73f00efc4861a6cf3005895ce4ee6b52
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.253@KRBTEST.COM" created.
I20260620 21:59:54.485509 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.469817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41609 (local address 127.0.0.1:47186)
0620 21:59:54.470420 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.470475 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.471474 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.472385 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.472390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.472931 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.473339 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.473362 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.474675 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.474681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.476042 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.476058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.476756 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.476763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.476946 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.477696 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.477720 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.479809 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.482089 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.482101 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.482117 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.482376 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.482868 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.482873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.482875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.482996 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.483647 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.483653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.483802 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.483990 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.484422 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":108,"spinlock_wait_cycles":22784,"thread_start_us":162,"threads_started":1}
I20260620 21:59:54.485509 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.469817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.17.67.254:41609)
0620 21:59:54.470371 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.470395 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.470588 (+ 193us) server_negotiation.cc:407] Connection header received
0620 21:59:54.471877 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.471886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.472143 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.472354 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.473526 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.474534 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.476188 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.476573 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.476698 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.479986 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.479992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.480004 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.480037 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.481954 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.482539 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.482550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.482554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.482755 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.483104 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.483108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.483110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.483528 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.483678 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.484053 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.484375 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":355,"thread_start_us":145,"threads_started":1}
I20260620 21:59:54.488566 7552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.489399 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.489116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34420)
0620 21:59:54.489245 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.489332 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 21:59:54.489738 7503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
I20260620 21:59:54.490657 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.490450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34434)
0620 21:59:54.490555 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.490604 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:59:54.490813 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:34215: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
I20260620 21:59:54.500174 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:34215
--webserver_interface=127.17.67.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:54.540058 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } attempt: 1
I20260620 21:59:54.540997 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.540711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34450)
0620 21:59:54.540850 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.540920 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:59:54.541677 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.541453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34464)
0620 21:59:54.541555 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.541628 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 21:59:54.541811 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:34215: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
I20260620 21:59:54.609965 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } attempt: 2
I20260620 21:59:54.610764 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.610481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34474)
0620 21:59:54.610623 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.610690 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 21:59:54.611534 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.611262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34486)
0620 21:59:54.611365 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.611457 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:59:54.611959 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:34215: Network error: Client connection negotiation failed: client connection to 127.17.67.253:34215: connect: Connection refused (error 111)
W20260620 21:59:54.612743 7559 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:54.613049 7559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:54.613108 7559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:54.613188 7559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:54.617241 7559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:54.617336 7559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:54.617377 7559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:54.617436 7559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:54.617477 7559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:54.623087 7559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.67.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:54.624277 7559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:54.625357 7559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:54.634140 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:54.634126 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.634205 7559 server_base.cc:1034] running on GCE node
W20260620 21:59:54.634130 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.635061 7559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:54.636216 7559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:54.637384 7559 hybrid_clock.cc:648] HybridClock initialized: now 1781992794637343 us; error 54 us; skew 500 ppm
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:54.640543 7559 init.cc:377] Logged in from keytab as kudu/127.17.67.253@KRBTEST.COM (short username kudu)
I20260620 21:59:54.641469 7559 webserver.cc:460] Webserver started at http://127.17.67.253:33461/ using document root <none> and password file <none>
I20260620 21:59:54.642067 7559 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:54.642165 7559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:54.642418 7559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:54.644517 7559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data/instance:
uuid: "92bffc82eefd4f64b606a8c00276d58d"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "46ce6b1df711892d705d75bcc07aa5bf"
server_key_iv: "cd3598e817a13be3aa74042810f4c06d"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.645104 7559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal/instance:
uuid: "92bffc82eefd4f64b606a8c00276d58d"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "46ce6b1df711892d705d75bcc07aa5bf"
server_key_iv: "cd3598e817a13be3aa74042810f4c06d"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.649026 7559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:59:54.651965 7574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:54.653362 7559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:59:54.653530 7559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "92bffc82eefd4f64b606a8c00276d58d"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "46ce6b1df711892d705d75bcc07aa5bf"
server_key_iv: "cd3598e817a13be3aa74042810f4c06d"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.653720 7559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:54.680222 7559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:54.683498 7559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:54.683727 7559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:54.690960 7559 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.253:34215
I20260620 21:59:54.691295 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.253:34215 every 8 connection(s)
I20260620 21:59:54.691941 7559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
I20260620 21:59:54.694782 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:54.697336 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7559
I20260620 21:59:54.697508 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-1/wal/instance
I20260620 21:59:54.697819 17679 external_mini_cluster.cc:1428] Setting key 6ce44137dd3ba3075a775f96ea508f95
I20260620 21:59:54.698741 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.703148 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } attempt: 3
WARNING: no policy specified for kudu/127.17.67.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.252@KRBTEST.COM" created.
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:54.715557 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.703560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34500)
0620 21:59:54.703680 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.703695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.703766 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.704546 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.704551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.704750 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.704883 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.704890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.706365 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.706368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.706956 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.706972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.707631 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.707764 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.708329 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.708341 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.710659 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.713206 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.713212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.713214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.713603 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.714095 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.714101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.714104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.714215 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.714990 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.714993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.715097 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.715222 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.715387 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 21:59:54.715960 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.700968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.17.67.254:41609)
0620 21:59:54.701143 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.701150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.701983 (+ 833us) server_negotiation.cc:407] Connection header received
0620 21:59:54.702941 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.702957 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.703031 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.703150 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.704980 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.705825 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707030 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.707296 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707372 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.710958 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.710962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.710965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.710993 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.713385 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.714350 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.714354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.714357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.714440 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.714754 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.714758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.714759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.714997 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.715099 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.715609 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.715783 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:59:54.716814 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.703689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.17.67.253:34215)
0620 21:59:54.704053 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.704064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.704088 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:59:54.704131 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.704136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.704368 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.704523 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.705029 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.706253 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707089 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.707520 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707621 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.710822 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.710832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.710847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.710879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.713067 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.713711 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.713717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.713719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.713972 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.714320 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.714322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.714323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.714886 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.715026 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.715378 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.715717 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":267,"spinlock_wait_cycles":5632,"thread_start_us":159,"threads_started":1}
I20260620 21:59:54.716817 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.701322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41609 (local address 127.0.0.1:47190)
0620 21:59:54.701818 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.701868 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.702709 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.703672 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.703688 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.704232 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.704848 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.704861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.705950 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.705957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.706917 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.706928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.707436 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.707441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.707603 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.708576 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.708597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.710818 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.713488 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.713497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.713507 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.713986 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.714540 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.714543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.714544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.714626 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.715159 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.715166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.715333 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.716087 (+ 754us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.716295 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":150,"thread_start_us":142,"threads_started":1}
I20260620 21:59:54.718298 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.719247 7552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.719982 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.719694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53816)
0620 21:59:54.719805 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.719885 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 21:59:54.720669 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.720402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53822)
0620 21:59:54.720563 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.720611 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 21:59:54.720836 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
I20260620 21:59:54.724658 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:33821
--webserver_interface=127.17.67.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:54.727036 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.718953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.17.67.253:34215)
0620 21:59:54.719280 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.719287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.719302 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:59:54.719382 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.719387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.719443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.719532 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.719997 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.720727 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.721472 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.721787 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.721895 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.723992 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.723995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.723997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.724017 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.725477 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.725924 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.725931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.725933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.726004 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.726264 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.726269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.726271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.726432 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.726529 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.726761 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.726900 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":246}
I20260620 21:59:54.727036 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.718850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34514)
0620 21:59:54.718945 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.718961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.719042 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.719568 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.719573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.719747 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.719887 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.719896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.720836 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.720839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.721364 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.721371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.721902 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.721905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.722043 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.722464 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.722475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.723872 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.725643 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.725646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.725648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.725829 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.726094 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.726099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.726101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.726179 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.726530 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.726533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.726615 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.726711 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.726849 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":19}
I20260620 21:59:54.728060 7627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.728801 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.728554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53830)
0620 21:59:54.728665 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.728745 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:59:54.729369 7578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
I20260620 21:59:54.730257 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.730055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53832)
0620 21:59:54.730156 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.730202 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 21:59:54.730424 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
I20260620 21:59:54.752020 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 1
I20260620 21:59:54.752949 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.752582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53838)
0620 21:59:54.752756 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.752848 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 21:59:54.753842 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.753597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53852)
0620 21:59:54.753708 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.753759 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 21:59:54.754073 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
I20260620 21:59:54.779707 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 1
I20260620 21:59:54.780694 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.780335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53868)
0620 21:59:54.780537 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.780602 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 21:59:54.781498 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.781282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53876)
0620 21:59:54.781393 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.781447 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:59:54.781648 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
I20260620 21:59:54.829267 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 2
I20260620 21:59:54.830397 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.829967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53892)
0620 21:59:54.830229 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.830304 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 21:59:54.831359 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.831077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53908)
0620 21:59:54.831190 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.831244 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:59:54.831512 7552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
W20260620 21:59:54.838555 7633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:54.838909 7633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:54.838972 7633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:54.839052 7633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:54.843350 7633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:54.843434 7633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:54.843482 7633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:54.843525 7633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:54.843561 7633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:54.849021 7633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:41609,127.17.67.253:34215,127.17.67.252:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.67.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:54.849855 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 2
I20260620 21:59:54.850235 7633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:54.850709 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.850422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53924)
0620 21:59:54.850560 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.850628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 21:59:54.851364 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.851150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53938)
0620 21:59:54.851267 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.851316 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 21:59:54.851374 7633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:54.851486 7627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:33821: Network error: Client connection negotiation failed: client connection to 127.17.67.252:33821: connect: Connection refused (error 111)
W20260620 21:59:54.859961 7639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:54.860015 7640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.860128 7633 server_base.cc:1034] running on GCE node
W20260620 21:59:54.859961 7642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:54.860862 7633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:54.862018 7633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:54.863204 7633 hybrid_clock.cc:648] HybridClock initialized: now 1781992794863165 us; error 49 us; skew 500 ppm
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:54.866420 7633 init.cc:377] Logged in from keytab as kudu/127.17.67.252@KRBTEST.COM (short username kudu)
I20260620 21:59:54.867458 7633 webserver.cc:460] Webserver started at http://127.17.67.252:38083/ using document root <none> and password file <none>
I20260620 21:59:54.868037 7633 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:54.868099 7633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:54.868355 7633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:54.870543 7633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data/instance:
uuid: "32096216d1af43f08a1830181d9831d1"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "7c907b0b8c8a2572e4694310202b7b63"
server_key_iv: "3e2d3a4fa8f83c27272d54eb163976ce"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.871131 7633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal/instance:
uuid: "32096216d1af43f08a1830181d9831d1"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "7c907b0b8c8a2572e4694310202b7b63"
server_key_iv: "3e2d3a4fa8f83c27272d54eb163976ce"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.874926 7633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:59:54.877377 7649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:54.878729 7633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:54.878880 7633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "32096216d1af43f08a1830181d9831d1"
format_stamp: "Formatted at 2026-06-20 21:59:54 on dist-test-slave-cx8d"
server_key: "7c907b0b8c8a2572e4694310202b7b63"
server_key_iv: "3e2d3a4fa8f83c27272d54eb163976ce"
server_key_version: "encryptionkey@0"
I20260620 21:59:54.878998 7633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:54.918201 7633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:54.921443 7633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:54.921689 7633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:54.928843 7633 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.252:33821
I20260620 21:59:54.929141 7701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.252:33821 every 8 connection(s)
I20260620 21:59:54.929934 7633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
I20260620 21:59:54.930686 7552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 3
I20260620 21:59:54.931874 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7633
I20260620 21:59:54.932003 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992513246304-17679-0/minicluster-data/master-2/wal/instance
I20260620 21:59:54.932405 17679 external_mini_cluster.cc:1428] Setting key 56ba5121a6a00f58ce43693a0a015149
I20260620 21:59:54.933719 7702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:54.938602 7702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:59:54.942643 7627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } attempt: 3
I20260620 21:59:54.945024 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.931184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53946)
0620 21:59:54.931335 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.931352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.931440 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.934322 (+ 2882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.934326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.934540 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.934658 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.934666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.936048 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.936053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.936852 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.936864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.937854 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.937857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.937947 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.938633 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.938649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.940262 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.942924 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.942927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.942930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.943170 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.943585 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.943588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.943590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.943703 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.944487 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.944490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.944608 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.944701 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.944828 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":3}
I20260620 21:59:54.947068 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.933060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.17.67.252:33821)
0620 21:59:54.933618 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.933627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.933642 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:59:54.933691 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.933693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.934038 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.934366 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.935456 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.936203 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.937258 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.937710 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.937875 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.941411 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.941415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.941417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.941439 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.944106 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.944620 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.944623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.944626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.944704 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.944996 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.944999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.945001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.945187 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.945360 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.946131 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.946259 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":965,"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":11008,"thread_start_us":119,"threads_started":1}
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:54.947065 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.931998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.17.67.252:33821)
0620 21:59:54.932534 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.932557 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.932604 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:59:54.933541 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.933595 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.934037 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.934367 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.934863 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.935910 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.936988 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.937708 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.938202 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.940497 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.940503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.940519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.940550 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.942791 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.943295 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.943301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.943307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.943470 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.943813 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.943817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.943819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.944317 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.944436 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.945723 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.946031 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":322,"spinlock_wait_cycles":6144,"thread_start_us":147,"threads_started":1}
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:59:54.952466 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.940983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.17.67.254:41609)
0620 21:59:54.941094 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.941098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.941173 (+ 75us) server_negotiation.cc:407] Connection header received
0620 21:59:54.941419 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.941426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.941483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.941588 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.942539 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.943303 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.944487 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.944844 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.944924 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.948860 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.948864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.948866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.948893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.950643 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.951128 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.951131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.951133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.951228 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.951505 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.951509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.951511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.951732 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.951841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.952183 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.952297 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":48}
I20260620 21:59:54.952478 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.940628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:41609 (local address 127.0.0.1:47192)
0620 21:59:54.941045 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.941101 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.941266 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.941616 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.941622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.942240 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.942381 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.942389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.943410 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.943415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.944363 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.944373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.945544 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.945552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.945693 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.946161 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.946175 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.948710 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.950765 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.950781 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.950786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.951003 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.951323 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.951328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.951330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.951411 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.951840 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.951848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.951966 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.952108 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.952298 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
I20260620 21:59:54.952673 7552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: Bootstrap starting.
I20260620 21:59:54.953514 7702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.955556 7552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: Neither blocks nor log segments found. Creating new log.
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:54.956461 7552 log.cc:824] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:54.958905 7552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: No bootstrap required, opened a new log
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
I20260620 21:59:54.960976 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.950006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.17.67.254:41609)
0620 21:59:54.950303 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.950317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.950333 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:59:54.950468 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.950472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.950525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.950622 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.951405 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.952152 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.953218 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.953526 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.953661 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.956522 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.956526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.956529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.956555 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.958360 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.958901 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.958905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.958906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.958995 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.959340 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.959344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.959345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.959672 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.959845 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.960650 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.960782 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260620 21:59:54.963141 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.943063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53960)
0620 21:59:54.943255 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.943268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.943342 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.943999 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.944005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.944189 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.944305 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.944316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.945435 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.945438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.946530 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.946541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.947131 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.947135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.947220 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.947736 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.947747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.948873 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.960530 (+ 11657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.960536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.960539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.960992 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.961983 (+ 991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.961987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.961989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.962076 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.962543 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.962546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.962652 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.962822 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.962965 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 21:59:54.963200 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.943076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.17.67.252:33821)
0620 21:59:54.943725 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.943733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.943747 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:54.943780 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.943783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.943826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.944420 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.944537 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.945317 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.946672 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.946993 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.947590 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.949023 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.949026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.949031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.949056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.960392 (+ 11336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.961770 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.961777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.961779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.961884 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.962199 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.962202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.962204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.962420 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.962551 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.962894 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.963016 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"spinlock_wait_cycles":4992,"thread_start_us":93,"threads_started":1}
I20260620 21:59:54.963445 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.954135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.17.67.253:34215)
0620 21:59:54.954262 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.954266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.954280 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:59:54.954320 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.954323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.954369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.954450 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.954976 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.955701 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.956820 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.957222 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.957318 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.959905 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.959909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.959912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.959941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.961728 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.962232 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.962235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.962237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.962326 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.962610 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.962613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.962614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.962807 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.962889 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.963202 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.963308 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:59:54.964298 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.953986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:34215 (local address 127.0.0.1:34528)
0620 21:59:54.954096 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.954111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.954193 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.954489 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.954494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.954693 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.954830 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.954839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.955863 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.955866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.956673 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.956684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.957507 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.957511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.957627 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.958032 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.958046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.959735 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.961838 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.961841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.961843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.962102 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.962444 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.962447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.962448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.962537 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.962936 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.962939 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.963024 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.964084 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.964197 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 21:59:54.964229 7552 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:54.964464 7552 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02a457552f04bc18269f31b9076e46a, State: Initialized, Role: FOLLOWER
I20260620 21:59:54.964829 7552 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:54.965029 7702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:54.965349 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:54.965519 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:54.966303 7552 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:54.966415 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: Bootstrap starting.
I20260620 21:59:54.970017 7627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:54.970930 7627 log.cc:824] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:54.973194 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: No bootstrap required, opened a new log
I20260620 21:59:54.971674 7484 master_runner.cc:186] Error getting master registration for 127.17.67.253:34215: OK, instance_id { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" instance_seqno: 1781992794688899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:59:54.975764 7484 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Jun 20 21:59:54 dist-test-slave-cx8d krb5kdc[7471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:59:54.977366 7627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:54.977624 7627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bffc82eefd4f64b606a8c00276d58d, State: Initialized, Role: FOLLOWER
I20260620 21:59:54.978003 7627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
W20260620 21:59:54.979095 7723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:54.979174 7723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:54.979456 7627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:54.982573 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.965505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:33821 (local address 127.0.0.1:53974)
0620 21:59:54.965756 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:54.965775 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:54.965848 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:54.971656 (+ 5808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:54.971661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:54.971849 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:54.971965 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.971972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.972923 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.972927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.973678 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:54.973689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.974252 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.974255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:54.974329 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.974850 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:54.974864 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:54.977051 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:54.981078 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.981082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.981084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.981298 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.981574 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:54.981577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:54.981579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:54.981656 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:54.982111 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:54.982114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:54.982195 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 21:59:54.982284 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:59:54.982409 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 21:59:54.982869 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.962262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.17.67.253:34215)
0620 21:59:54.963593 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.963597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.963607 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:54.963634 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.963636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.963691 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.963766 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.969413 (+ 5647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.970160 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.971264 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.971587 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.971683 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.977804 (+ 6121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.977807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.977810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.977838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.980007 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.980604 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.980607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.980609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.980696 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.981023 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.981029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.981034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.981290 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.981492 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.982291 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.982406 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":132,"threads_started":1}
I20260620 21:59:54.983664 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.965731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.17.67.252:33821)
0620 21:59:54.971426 (+ 5695us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.971432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.971449 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:59:54.971490 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.971492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.971538 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.972064 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.972124 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.972807 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.973804 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.974119 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:54.974527 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:54.977204 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:54.977208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:54.977210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:54.977234 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:54.980968 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.981394 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.981397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.981399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.981471 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:54.981758 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:54.981760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:54.981762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:54.981958 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:54.982074 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:54.982987 (+ 913us) server_negotiation.cc:299] Negotiation successful
0620 21:59:54.983096 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5615}
I20260620 21:59:54.986099 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:54.986755 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: Bootstrap starting.
I20260620 21:59:54.987345 7526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d02a457552f04bc18269f31b9076e46a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47200
I20260620 21:59:54.987561 7526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:59:54.987653 7526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:54.988992 7526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:54.989542 7702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:54.990031 7526 leader_election.cc:288] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [CANDIDATE]: Term 1 election: Requested vote from peers 92bffc82eefd4f64b606a8c00276d58d (127.17.67.253:34215), 32096216d1af43f08a1830181d9831d1 (127.17.67.252:33821)
I20260620 21:59:54.990448 7702 log.cc:824] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:54.990764 7601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d02a457552f04bc18269f31b9076e46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92bffc82eefd4f64b606a8c00276d58d"
I20260620 21:59:54.990993 7601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:54.987486 7559 master_runner.cc:186] Error getting master registration for 127.17.67.252:33821: OK, instance_id { permanent_uuid: "32096216d1af43f08a1830181d9831d1" instance_seqno: 1781992794926874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:59:54.991329 7559 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:59:54.991973 7601 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02a457552f04bc18269f31b9076e46a in term 1.
I20260620 21:59:54.992388 7503 leader_election.cc:302] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bffc82eefd4f64b606a8c00276d58d, d02a457552f04bc18269f31b9076e46a; no voters:
I20260620 21:59:54.992674 7712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:54.992899 7712 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [term 1 LEADER]: Becoming Leader. State: Replica: d02a457552f04bc18269f31b9076e46a, State: Running, Role: LEADER
I20260620 21:59:54.993223 7712 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
W20260620 21:59:54.994952 7737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:54.995026 7737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:54.995098 7726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:54.995113 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02a457552f04bc18269f31b9076e46a. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:54.995203 7726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:54.995219 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:54.995609 7746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:54.996649 7746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:55.002612 7601 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [term 1 FOLLOWER]: Refusing update from remote peer d02a457552f04bc18269f31b9076e46a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:55.003515 7712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:55.005035 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: No bootstrap required, opened a new log
I20260620 21:59:55.008117 7726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02a457552f04bc18269f31b9076e46a. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.008442 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.995590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.17.67.254:41609)
0620 21:59:54.995708 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.995714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:54.996114 (+ 400us) server_negotiation.cc:407] Connection header received
0620 21:59:54.996437 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:54.996440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:54.996491 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:54.996544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:54.998191 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:54.998943 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.000575 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:55.000905 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.000980 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:55.003868 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:55.003883 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:55.003886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:55.003914 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:55.005809 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.006606 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.006611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.006612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.006704 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.007182 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.007186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.007187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.007414 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:55.007526 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:55.008093 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 21:59:55.008263 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:59:55.008708 7726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:55.010015 7702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:55.010296 7702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32096216d1af43f08a1830181d9831d1, State: Initialized, Role: FOLLOWER
I20260620 21:59:55.010668 7702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } }
I20260620 21:59:55.011260 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.011468 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:55.012069 7702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:55.012329 7675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:55.013732 7675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [term 1 FOLLOWER]: Refusing update from remote peer d02a457552f04bc18269f31b9076e46a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:55.014470 7712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:55.015678 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.015789 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:55.015966 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d02a457552f04bc18269f31b9076e46a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32096216d1af43f08a1830181d9831d1"
I20260620 21:59:55.017364 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.996157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.17.67.253:34215)
0620 21:59:54.996262 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.996267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.001822 (+ 5555us) server_negotiation.cc:407] Connection header received
0620 21:59:55.002014 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.002017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.002071 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.002128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.006197 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:55.006944 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.008195 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:55.008538 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.008621 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:55.013666 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:55.013671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:55.013673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:55.013709 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:55.015492 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.015994 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.015998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.015999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.016086 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.016373 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.016376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.016378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.016591 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:55.016714 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:55.017074 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 21:59:55.017201 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:59:55.017714 7676 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02a457552f04bc18269f31b9076e46a in term 1.
I20260620 21:59:55.019151 7751 mvcc.cc:205] Tried to move back new op lower bound from 7299042488321470464 to 7299042488295989248. Current Snapshot: MvccSnapshot[applied={T|T < 7299042488321470464}]
I20260620 21:59:55.019444 7726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.019536 7726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:55.022197 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02a457552f04bc18269f31b9076e46a. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.022318 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:59:55.023278 7764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 21:59:55.024724 7746 catalog_manager.cc:1347] Generated new cluster ID: bb1b16a6ddfb40d5b1e9dc95b893ec52
I20260620 21:59:55.024794 7746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260620 21:59:55.024859 7764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:55.030296 7739 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.030447 7739 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:55.031697 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d02a457552f04bc18269f31b9076e46a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02a457552f04bc18269f31b9076e46a" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 41609 } } peers { permanent_uuid: "92bffc82eefd4f64b606a8c00276d58d" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 34215 } } peers { permanent_uuid: "32096216d1af43f08a1830181d9831d1" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 33821 } } }
I20260620 21:59:55.031840 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:55.031984 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:54.996740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.17.67.252:33821)
0620 21:59:54.996842 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:54.996846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.009750 (+ 12904us) server_negotiation.cc:407] Connection header received
0620 21:59:55.009926 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.009929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.009974 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.010027 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.010995 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:55.011691 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.025694 (+ 14003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:55.026054 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:55.026142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:55.028144 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:55.028148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:55.028152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:55.028208 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:55.029981 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.030511 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.030515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.030516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.030603 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:55.030935 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:55.030938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:55.030940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:55.031156 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:55.031286 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:55.031673 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 21:59:55.031797 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:59:55.032318 7768 mvcc.cc:205] Tried to move back new op lower bound from 7299042488321470464 to 7299042488295989248. Current Snapshot: MvccSnapshot[applied={T|T < 7299042488321470464}]
I20260620 21:59:55.037751 7746 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:55.040771 7746 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:55.053632 7746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02a457552f04bc18269f31b9076e46a: Generated new TSK 0
I20260620 21:59:55.054527 7746 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:59:55.080139 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.077123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.17.67.253:34215)
0620 21:59:55.077363 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.077371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.078834 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 21:59:55.079247 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.079251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.079303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.079356 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.079991 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34564: BlockingRecv error: recv got EOF from 127.0.0.1:34564 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 21:59:55.081239 7708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.078172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.17.67.252:33821)
0620 21:59:55.078297 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.078301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.079150 (+ 849us) server_negotiation.cc:407] Connection header received
0620 21:59:55.080486 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.080490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.080540 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.080593 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.081107 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53994: BlockingRecv error: recv got EOF from 127.0.0.1:53994 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 21:59:55.083364 7709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.082027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.17.67.254:41609)
0620 21:59:55.082185 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.082189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.082505 (+ 316us) server_negotiation.cc:407] Connection header received
0620 21:59:55.082672 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.082676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.082722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.082770 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.083250 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47216: BlockingRecv error: recv got EOF from 127.0.0.1:47216 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 21:59:55.087759 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.086189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.17.67.253:34215)
0620 21:59:55.086292 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.086296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.086797 (+ 501us) server_negotiation.cc:407] Connection header received
0620 21:59:55.087016 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.087020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.087098 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.087149 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.087653 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34572: BlockingRecv error: recv got EOF from 127.0.0.1:34572 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 21:59:55.092798 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.091239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.17.67.253:34215)
0620 21:59:55.091347 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.091351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.091889 (+ 538us) server_negotiation.cc:407] Connection header received
0620 21:59:55.092077 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.092081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.092132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.092195 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.092685 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34584: BlockingRecv error: recv got EOF from 127.0.0.1:34584 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 21:59:55.098263 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.096698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.17.67.253:34215)
0620 21:59:55.096796 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.096800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.097329 (+ 529us) server_negotiation.cc:407] Connection header received
0620 21:59:55.097511 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.097514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.097612 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.097665 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.098148 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34592: BlockingRecv error: recv got EOF from 127.0.0.1:34592 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 21:59:55.105863 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.104222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.17.67.253:34215)
0620 21:59:55.104369 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.104373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.104900 (+ 527us) server_negotiation.cc:407] Connection header received
0620 21:59:55.105084 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.105088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.105140 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.105206 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.105755 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34600: BlockingRecv error: recv got EOF from 127.0.0.1:34600 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 21:59:55.117753 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.116019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.17.67.253:34215)
0620 21:59:55.116133 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.116137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.116695 (+ 558us) server_negotiation.cc:407] Connection header received
0620 21:59:55.116895 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.116898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.116950 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.117002 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.117634 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34608: BlockingRecv error: recv got EOF from 127.0.0.1:34608 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:59:55.138085 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.136232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.17.67.253:34215)
0620 21:59:55.136358 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.136363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.136998 (+ 635us) server_negotiation.cc:407] Connection header received
0620 21:59:55.137231 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.137235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.137293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.137349 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.137965 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34622: BlockingRecv error: recv got EOF from 127.0.0.1:34622 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 21:59:55.174710 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.172680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.17.67.253:34215)
0620 21:59:55.172821 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.172826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.173630 (+ 804us) server_negotiation.cc:407] Connection header received
0620 21:59:55.173868 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.173872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.173928 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.173984 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.174583 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34638: BlockingRecv error: recv got EOF from 127.0.0.1:34638 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 21:59:55.242753 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.241402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.17.67.253:34215)
0620 21:59:55.241596 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.241601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.241826 (+ 225us) server_negotiation.cc:407] Connection header received
0620 21:59:55.241993 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.241995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.242044 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.242132 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.242582 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34650: BlockingRecv error: recv got EOF from 127.0.0.1:34650 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 21:59:55.374004 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.372870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.17.67.253:34215)
0620 21:59:55.373056 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.373061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.373078 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:59:55.373232 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.373234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.373288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.373367 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.373867 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34654: BlockingRecv error: recv got EOF from 127.0.0.1:34654 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 21:59:55.633507 7631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:55.632202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.17.67.253:34215)
0620 21:59:55.632357 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:55.632363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:55.632480 (+ 117us) server_negotiation.cc:407] Connection header received
0620 21:59:55.632705 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:55.632708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:55.632759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:55.632868 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:55.633392 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34668: BlockingRecv error: recv got EOF from 127.0.0.1:34668 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:59:55.996371 7737 catalog_manager.cc:1259] Loaded cluster ID: bb1b16a6ddfb40d5b1e9dc95b893ec52
I20260620 21:59:55.996460 7737 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: loading cluster ID for follower catalog manager: success
I20260620 21:59:55.998183 7737 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: acquiring CA information for follower catalog manager: success
I20260620 21:59:55.999382 7737 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92bffc82eefd4f64b606a8c00276d58d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:59:56.026325 7764 catalog_manager.cc:1259] Loaded cluster ID: bb1b16a6ddfb40d5b1e9dc95b893ec52
I20260620 21:59:56.026415 7764 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: loading cluster ID for follower catalog manager: success
I20260620 21:59:56.028123 7764 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: acquiring CA information for follower catalog manager: success
I20260620 21:59:56.029340 7764 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32096216d1af43f08a1830181d9831d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:59:56.151428 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:56.147708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.17.67.253:34215)
0620 21:59:56.147989 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:56.147993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:56.148053 (+ 60us) server_negotiation.cc:407] Connection header received
0620 21:59:56.148184 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:56.148186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:56.148320 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:56.148421 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:56.148641 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.149169 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.150009 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.150238 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.150329 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:56.150578 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:56.151020 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:56.151114 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:56.151172 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 21:59:56.151220 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260620 21:59:56.158627 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:56.154889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.17.67.252:33821)
0620 21:59:56.155130 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:56.155137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:56.155168 (+ 31us) server_negotiation.cc:407] Connection header received
0620 21:59:56.155259 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:56.155262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:56.155372 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:56.155495 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:56.155725 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.156257 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.157096 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.157339 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.157453 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:56.157820 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:56.158233 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:56.158370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:56.158411 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 21:59:56.158467 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
I20260620 21:59:56.166425 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:56.162693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.17.67.252:33821)
0620 21:59:56.162837 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:56.162842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:56.163136 (+ 294us) server_negotiation.cc:407] Connection header received
0620 21:59:56.163251 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:56.163255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:56.163299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:56.163402 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:56.163737 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.164267 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.165099 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.165338 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.165446 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:56.165758 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:56.165877 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:56.165950 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:56.166231 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 21:59:56.166279 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:59:56.166441 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:56.162059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.17.67.253:34215)
0620 21:59:56.162214 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:56.162219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:56.162326 (+ 107us) server_negotiation.cc:407] Connection header received
0620 21:59:56.162551 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:56.162555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:56.162621 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:56.162675 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:56.163086 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.163735 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.165020 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.165401 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.165479 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:56.165891 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:56.165971 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:56.166060 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:56.166212 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 21:59:56.166277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:59:56.167430 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:56.162110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.17.67.254:41609)
0620 21:59:56.162461 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:56.162468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:56.162522 (+ 54us) server_negotiation.cc:407] Connection header received
0620 21:59:56.162722 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:56.162726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:56.162899 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:56.163023 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:56.163397 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.164296 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.165614 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:56.165967 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:56.166075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:56.166476 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:56.166955 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:56.167104 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:56.167197 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 21:59:56.167261 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
I20260620 21:59:56.170751 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7484
I20260620 21:59:56.177443 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7559
I20260620 21:59:56.183007 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7633
2026-06-20T21:59:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3113 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:59:56 dist-test-slave-cx8d krb5kdc[7853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:59:56 dist-test-slave-cx8d krb5kdc[7853](info): set up 2 sockets
Jun 20 21:59:56 dist-test-slave-cx8d krb5kdc[7853](info): commencing operation
krb5kdc: starting...
W20260620 21:59:57.221961 17679 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/test-user.keytab.
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.67.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
Entry for principal kudu/127.17.67.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab.
I20260620 21:59:57.340564 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39935
--webserver_interface=127.17.67.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:59:57.452973 7866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:57.453389 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:57.453459 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:57.453557 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:57.457664 7866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:57.457746 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:57.457783 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:57.457823 7866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:57.457858 7866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:57.463058 7866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.254.keytab
--principal=kudu/127.17.67.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.254:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.67.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:57.464097 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:57.465445 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:57.474670 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:57.474797 7866 server_base.cc:1034] running on GCE node
W20260620 21:59:57.474680 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:57.474994 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:57.475576 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:57.476754 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:57.477955 7866 hybrid_clock.cc:648] HybridClock initialized: now 1781992797477938 us; error 29 us; skew 500 ppm
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:57.480973 7866 init.cc:377] Logged in from keytab as kudu/127.17.67.254@KRBTEST.COM (short username kudu)
I20260620 21:59:57.481992 7866 webserver.cc:460] Webserver started at http://127.17.67.254:42331/ using document root <none> and password file <none>
I20260620 21:59:57.482549 7866 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:57.482661 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:57.482908 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:57.484926 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data/instance:
uuid: "02cce4b8d970499c86c58a659c85c2fe"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "2b52dda57056466e4615792a01f3b81d"
server_key_iv: "6f5f2e2b8e7692e65e29d216761291cb"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.485495 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance:
uuid: "02cce4b8d970499c86c58a659c85c2fe"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "2b52dda57056466e4615792a01f3b81d"
server_key_iv: "6f5f2e2b8e7692e65e29d216761291cb"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.489223 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:59:57.491802 7881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:57.493182 7866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:59:57.493343 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal
uuid: "02cce4b8d970499c86c58a659c85c2fe"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "2b52dda57056466e4615792a01f3b81d"
server_key_iv: "6f5f2e2b8e7692e65e29d216761291cb"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.493475 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:57.505618 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:57.509001 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:57.509215 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:57.516188 7866 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.254:39935
I20260620 21:59:57.516489 7933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.254:39935 every 8 connection(s)
I20260620 21:59:57.517171 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/data/info.pb
I20260620 21:59:57.519912 7934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:57.523653 7934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.527088 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7866
I20260620 21:59:57.527215 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-0/wal/instance
I20260620 21:59:57.527479 17679 external_mini_cluster.cc:1428] Setting key 0178f78f5a7c6c446c3f53002bd99237
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.253@KRBTEST.COM" created.
I20260620 21:59:57.538314 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.525932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.17.67.254:39935)
0620 21:59:57.526349 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.526365 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.526591 (+ 226us) server_negotiation.cc:407] Connection header received
0620 21:59:57.527666 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.527669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.527863 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.528046 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.528917 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.529834 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.530805 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.531160 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.531324 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.533644 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.533649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.533659 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.533678 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.535293 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.535733 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.535741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.535747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.535848 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.536153 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.536156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.536158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.536507 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.536644 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.537003 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.537239 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":110,"threads_started":1}
I20260620 21:59:57.538314 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.525932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39935 (local address 127.0.0.1:45872)
0620 21:59:57.526451 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.526489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.527363 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.528044 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.528049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.528522 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.528752 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.528768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.529959 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.529962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.530680 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.530688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.531312 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.531316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.531422 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.532035 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.532050 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.533477 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.535394 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.535403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.535413 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.535625 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.535946 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.535949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.535951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.536058 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.536625 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.536631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.536771 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.536933 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.537318 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":103,"spinlock_wait_cycles":17152,"thread_start_us":192,"threads_started":1}
I20260620 21:59:57.540846 7934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.541543 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.541292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52688)
0620 21:59:57.541407 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.541482 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:59:57.541747 7884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
I20260620 21:59:57.542584 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.542399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52698)
0620 21:59:57.542496 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.542538 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:59:57.542820 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:42191: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
Entry for principal kudu/127.17.67.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab.
I20260620 21:59:57.554061 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:42191
--webserver_interface=127.17.67.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:57.592106 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } attempt: 1
I20260620 21:59:57.593111 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.592782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52704)
0620 21:59:57.592962 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.593030 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 21:59:57.594179 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.593924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52708)
0620 21:59:57.594058 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.594111 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 21:59:57.594424 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:42191: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
I20260620 21:59:57.662635 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } attempt: 2
I20260620 21:59:57.663477 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.663145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52722)
0620 21:59:57.663331 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.663397 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 21:59:57.664322 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.664070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52738)
0620 21:59:57.664180 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.664229 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:59:57.664503 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.253:42191: Network error: Client connection negotiation failed: client connection to 127.17.67.253:42191: connect: Connection refused (error 111)
W20260620 21:59:57.667846 7941 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:57.668109 7941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:57.668188 7941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:57.668272 7941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:57.672549 7941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:57.672613 7941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:57.672633 7941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:57.672650 7941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:57.672667 7941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:57.678105 7941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.253.keytab
--principal=kudu/127.17.67.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.253:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.67.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:57.679160 7941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:57.680219 7941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:57.688803 7949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:57.688784 7946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:57.689011 7941 server_base.cc:1034] running on GCE node
W20260620 21:59:57.688827 7947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:57.689796 7941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:57.690790 7941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:57.691948 7941 hybrid_clock.cc:648] HybridClock initialized: now 1781992797691933 us; error 29 us; skew 500 ppm
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:57.694748 7941 init.cc:377] Logged in from keytab as kudu/127.17.67.253@KRBTEST.COM (short username kudu)
I20260620 21:59:57.695600 7941 webserver.cc:460] Webserver started at http://127.17.67.253:34737/ using document root <none> and password file <none>
I20260620 21:59:57.696149 7941 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:57.696208 7941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:57.696444 7941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:57.698586 7941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data/instance:
uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "e27c2b133db753f7c80f9a5d9e8fedcd"
server_key_iv: "328bfadb700a3a2f7b0e4b8788b08e20"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.699150 7941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal/instance:
uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "e27c2b133db753f7c80f9a5d9e8fedcd"
server_key_iv: "328bfadb700a3a2f7b0e4b8788b08e20"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.702865 7941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:59:57.705309 7956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:57.706671 7941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:59:57.706848 7941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal
uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "e27c2b133db753f7c80f9a5d9e8fedcd"
server_key_iv: "328bfadb700a3a2f7b0e4b8788b08e20"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.706972 7941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:57.719300 7941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:57.722478 7941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:57.722689 7941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:57.730086 7941 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.253:42191
I20260620 21:59:57.730427 8008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.253:42191 every 8 connection(s)
I20260620 21:59:57.731160 7941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/data/info.pb
I20260620 21:59:57.733999 8009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:57.737811 8009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.740634 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 7941
I20260620 21:59:57.740783 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-1/wal/instance
I20260620 21:59:57.741103 17679 external_mini_cluster.cc:1428] Setting key c8560139179d79dde225b077b4a5c7e7
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.67.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.67.252@KRBTEST.COM" created.
I20260620 21:59:57.751942 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.739808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.17.67.254:39935)
0620 21:59:57.739995 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.740000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.740976 (+ 976us) server_negotiation.cc:407] Connection header received
0620 21:59:57.742065 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.742069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.742136 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.742240 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.743451 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.743976 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.745156 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.745404 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.745523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.748018 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.748022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.748024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.748048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.749688 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.750211 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.750215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.750217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.750306 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.750668 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.750673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.750675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.750922 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.751040 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.751620 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.751751 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:59:57.752802 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.740189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39935 (local address 127.0.0.1:45884)
0620 21:59:57.740771 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.740836 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.741843 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.742420 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.742428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.742799 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.743296 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.743309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.744109 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.744113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.745023 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.745032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.745622 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.745628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.745791 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.746406 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.746422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.747832 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.749811 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.749818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.749833 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.750102 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.750418 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.750422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.750424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.750570 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.751100 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.751105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.751353 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.751566 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.751909 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":191,"threads_started":1}
I20260620 21:59:57.754487 8009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.755692 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } attempt: 3
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
Entry for principal kudu/127.17.67.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab.
I20260620 21:59:57.766957 17679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
/tmp/dist-test-taskiPMPYE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:46203
--webserver_interface=127.17.67.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.67.212:45827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:59:57.767123 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.756102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52756)
0620 21:59:57.756270 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.756284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.756353 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.756788 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.756792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.756977 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.757067 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.757073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.758157 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.758162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.759110 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.759120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.759749 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.759862 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.760411 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.760423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.762318 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.765228 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.765234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.765239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.765502 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.766025 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.766029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.766030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.766125 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.766552 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.766555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.766657 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.766779 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.766933 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 21:59:57.767185 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.755068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52752)
0620 21:59:57.755521 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.755533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.755666 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.756302 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.756308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.756505 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.756624 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.756633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.758214 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.758218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.759140 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.759153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.759896 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.759961 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.760575 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.760586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.762734 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.765300 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.765306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.765310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.765502 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.765897 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.765899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.765901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.766018 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.766608 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.766614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.766710 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.766817 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.766964 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260620 21:59:57.767346 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.756172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.17.67.253:42191)
0620 21:59:57.756595 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.756604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.756616 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:57.756644 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.756647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.756689 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.756770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.757183 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.758046 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759238 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.759625 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759749 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.762492 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.762503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.762519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.762545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.765112 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.765639 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.765646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.765649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.765823 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.766234 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.766241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.766245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.766434 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.766573 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.766882 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.767019 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":84736,"thread_start_us":104,"threads_started":1}
I20260620 21:59:57.767763 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.755169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.17.67.253:42191)
0620 21:59:57.755467 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.755480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.755615 (+ 135us) server_negotiation.cc:407] Connection header received
0620 21:59:57.755773 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.755779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.756026 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.756159 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.756759 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.758084 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759266 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.759656 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.759767 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.762956 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.762963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.762969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.763001 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.765140 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.765639 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.765647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.765652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.765794 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.766116 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.766118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.766120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.766444 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.766573 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.767373 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.767499 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":200,"spinlock_wait_cycles":1920,"thread_start_us":124,"threads_started":1}
I20260620 21:59:57.769306 8009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.769622 7934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:59:57.770112 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.769800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50432)
0620 21:59:57.769941 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.770033 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:59:57.770332 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.770040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50444)
0620 21:59:57.770203 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.770264 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:59:57.770344 7959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.771238 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.770963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50460)
0620 21:59:57.771102 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.771163 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:59:57.771435 8009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.771656 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.771406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50462)
0620 21:59:57.771551 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.771595 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 21:59:57.771821 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.803030 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 1
I20260620 21:59:57.803943 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.803558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50478)
0620 21:59:57.803797 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.803865 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 21:59:57.804790 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.804559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50486)
0620 21:59:57.804683 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.804733 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 21:59:57.804948 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.820708 8009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 1
I20260620 21:59:57.821591 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.821251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50498)
0620 21:59:57.821405 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.821475 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 21:59:57.822356 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.822135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50500)
0620 21:59:57.822247 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.822298 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:59:57.822507 8009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.880112 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 2
I20260620 21:59:57.880892 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.880618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50502)
0620 21:59:57.880752 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.880817 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 21:59:57.881671 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.881417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50508)
0620 21:59:57.881522 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.881610 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:59:57.881501 8016 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:59:57.881812 7934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
W20260620 21:59:57.881865 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:59:57.881912 8016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:59:57.881981 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:59:57.886011 8016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:59:57.886063 8016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:59:57.886082 8016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:59:57.886101 8016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:59:57.886140 8016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:59:57.890661 8009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 2
I20260620 21:59:57.891394 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.891104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50512)
0620 21:59:57.891247 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.891312 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 21:59:57.891739 8016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.67.212:45827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.67.254:39935,127.17.67.253:42191,127.17.67.252:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/krb5kdc/kudu_127.17.67.252.keytab
--principal=kudu/127.17.67.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.67.252:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.67.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:59:57.892241 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.891912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50524)
0620 21:59:57.892132 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.892181 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260620 21:59:57.892390 8009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.67.252:46203: Network error: Client connection negotiation failed: client connection to 127.17.67.252:46203: connect: Connection refused (error 111)
I20260620 21:59:57.892935 8016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:59:57.894052 8016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:59:57.902294 8025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:57.902294 8022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:59:57.902305 8023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:59:57.902693 8016 server_base.cc:1034] running on GCE node
I20260620 21:59:57.903254 8016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:59:57.904295 8016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:59:57.905457 8016 hybrid_clock.cc:648] HybridClock initialized: now 1781992797905443 us; error 31 us; skew 500 ppm
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:59:57.908376 8016 init.cc:377] Logged in from keytab as kudu/127.17.67.252@KRBTEST.COM (short username kudu)
I20260620 21:59:57.909207 8016 webserver.cc:460] Webserver started at http://127.17.67.252:35825/ using document root <none> and password file <none>
I20260620 21:59:57.909783 8016 fs_manager.cc:359] Metadata directory not provided
I20260620 21:59:57.909847 8016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:59:57.910019 8016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:59:57.912094 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data/instance:
uuid: "5f16a15a48f043b38bdc25e84f6b71d8"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "248f29824fcbb7dbc99b7b05877a047f"
server_key_iv: "d894f9a6774dc2e395db87afb3782d02"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.912599 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal/instance:
uuid: "5f16a15a48f043b38bdc25e84f6b71d8"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "248f29824fcbb7dbc99b7b05877a047f"
server_key_iv: "d894f9a6774dc2e395db87afb3782d02"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.916107 8016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 21:59:57.918380 8032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:59:57.919670 8016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:59:57.919766 8016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data,/tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal
uuid: "5f16a15a48f043b38bdc25e84f6b71d8"
format_stamp: "Formatted at 2026-06-20 21:59:57 on dist-test-slave-cx8d"
server_key: "248f29824fcbb7dbc99b7b05877a047f"
server_key_iv: "d894f9a6774dc2e395db87afb3782d02"
server_key_version: "encryptionkey@0"
I20260620 21:59:57.919848 8016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:59:57.932291 8016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:59:57.935279 8016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:59:57.935436 8016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:59:57.942226 8016 rpc_server.cc:307] RPC server started. Bound to: 127.17.67.252:46203
I20260620 21:59:57.942550 8084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.67.252:46203 every 8 connection(s)
I20260620 21:59:57.943213 8016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/data/info.pb
I20260620 21:59:57.943878 17679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu as pid 8016
I20260620 21:59:57.944067 17679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiPMPYE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992513246304-17679-0/minicluster-data/master-2/wal/instance
I20260620 21:59:57.944394 17679 external_mini_cluster.cc:1428] Setting key 0ea503a865e19df1e3b1512fad502e55
I20260620 21:59:57.946390 8085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:59:57.950543 8085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
I20260620 21:59:57.959093 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.945818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.17.67.252:46203)
0620 21:59:57.946231 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.946247 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.946281 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:59:57.946966 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.946982 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.947308 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.947587 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.948502 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.949516 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.950719 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.951235 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.951379 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.954224 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.954228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.954237 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.954262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.956202 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.956678 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.956683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.956687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.956826 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.957103 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.957108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.957112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.957656 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.957802 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.958101 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.958390 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260620 21:59:57.962587 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.952396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.254:39935 (local address 127.0.0.1:45886)
0620 21:59:57.952797 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.952855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.952991 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.953310 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.953315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.953709 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.953883 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.953889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.954812 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.954815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.955891 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.955902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.956371 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.956373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.956453 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.956890 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.956907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.958569 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.960500 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.960508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.960513 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.960844 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.961221 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.961225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.961226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.961325 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.961979 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.961986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.962112 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.962265 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.962434 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260620 21:59:57.962739 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.952490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.17.67.254:39935)
0620 21:59:57.952618 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.952622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.952950 (+ 328us) server_negotiation.cc:407] Connection header received
0620 21:59:57.953101 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.953105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.953166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.953259 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.954014 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.954637 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.956022 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.956228 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.956396 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.958709 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.958712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.958715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.958736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.960376 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.961032 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.961034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.961036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.961113 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.961527 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.961531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.961533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.961861 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.961964 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.962368 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.962521 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:59:57.963979 8085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.254@KRBTEST.COM
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
I20260620 21:59:57.971537 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.961545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.17.67.254:39935)
0620 21:59:57.962005 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.962015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.962026 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:59:57.962061 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.962064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.962106 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.962200 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.963078 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.963781 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.964930 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.965209 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.965334 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.968046 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.968049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.968051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.968075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.969743 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.970215 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.970220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.970222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.970308 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.970609 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.970613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.970615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.970805 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.971021 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.971196 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.971337 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":151,"threads_started":1}
I20260620 21:59:57.972795 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.964447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.253:42191 (local address 127.0.0.1:52766)
0620 21:59:57.964553 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.964568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.964652 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.964948 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.964951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.965119 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.965206 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.965211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.966149 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.966152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.966807 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.966816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.967283 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.967286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.967388 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.967875 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.967888 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.969293 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.971100 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.971105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.971107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.971301 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.971741 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.971744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.971746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.971848 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.972320 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.972325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.972417 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.972539 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.972650 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 21:59:57.973137 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.964585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.17.67.253:42191)
0620 21:59:57.964744 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.964749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.964762 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:59:57.964807 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.964809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.964851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.964936 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.965323 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.966028 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.966925 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.967174 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.967282 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.969461 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.969464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.969467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.969493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.970986 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.971536 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.971538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.971539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.971647 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.971978 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.971981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.971983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.972198 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.972300 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.972603 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.972731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:59:57.973830 8085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.67.253@KRBTEST.COM
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.252@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:59:57.980970 7934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 3
I20260620 21:59:57.982755 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.972969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.17.67.253:42191)
0620 21:59:57.973109 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.973112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.973122 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:59:57.973149 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.973151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.973209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.973292 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.974110 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.974888 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.975754 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.976051 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.976211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.978710 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.978713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.978715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.978734 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.980467 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.980963 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.980966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.980968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.981062 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.981725 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.981728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.981730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.981968 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.982178 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.982419 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.982549 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:59:57.983417 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.974306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50548)
0620 21:59:57.974464 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.974476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.974540 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.974741 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.974743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.974898 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.975002 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.975008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.975697 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.975700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.976184 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.976191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.976757 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.976761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.976875 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.977293 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.977306 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.979045 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.981363 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.981366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.981367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.981621 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.981946 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.981949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.981950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.982040 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.982492 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.982497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.982601 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.982710 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.983227 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5}
I20260620 21:59:57.983439 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.974377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.17.67.252:46203)
0620 21:59:57.974492 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.974496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.974515 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:59:57.974614 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.974616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.974647 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.974713 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.975104 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.975588 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.976291 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.976648 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.976760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.979175 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.979180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.979184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.979211 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.981235 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.981702 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.981706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.981708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.981826 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.982142 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.982145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.982147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.982369 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.982458 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.982773 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.983298 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":430,"server-negotiator.queue_time_us":47}
I20260620 21:59:57.983656 8009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } attempt: 3
I20260620 21:59:57.986943 17679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:59:57.987329 8085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: Bootstrap starting.
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.254@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
Jun 20 21:59:57 dist-test-slave-cx8d krb5kdc[7853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.67.253@KRBTEST.COM for kudu/127.17.67.252@KRBTEST.COM
I20260620 21:59:57.988267 7908 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "02cce4b8d970499c86c58a659c85c2fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45888
I20260620 21:59:57.990335 8085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:57.991473 8085 log.cc:824] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:57.994488 8085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: No bootstrap required, opened a new log
I20260620 21:59:57.994730 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.981378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50560)
0620 21:59:57.981507 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.981522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.981619 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.982320 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.982323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.982483 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.982556 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.982561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.984792 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.984796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.986081 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.986091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.986697 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.986701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.986790 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.987300 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.987314 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.989998 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.992957 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.992961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.992963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.993224 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.993530 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.993535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.993650 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.994074 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.994077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.994273 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.994419 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.994581 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 21:59:57.994910 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.984000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.67.252:46203 (local address 127.0.0.1:50568)
0620 21:59:57.984140 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:59:57.984153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:59:57.984233 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:59:57.984604 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:59:57.984607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:59:57.984783 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:59:57.984870 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.984876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.985820 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.985827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.986644 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:59:57.986655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.987276 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.987280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:59:57.987370 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.987797 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:59:57.987812 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:59:57.989629 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:59:57.992827 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.992831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.992833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.993117 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.993480 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:59:57.993485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:59:57.993620 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:59:57.994310 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:59:57.994313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:59:57.994433 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 21:59:57.994532 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 21:59:57.994691 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 21:59:57.995538 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.981678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.17.67.252:46203)
0620 21:59:57.982061 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.982071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.982083 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:59:57.982115 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.982118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.982155 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.982283 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.982664 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.983389 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.986220 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.986572 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.990137 (+ 3565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.990251 (+ 114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.990266 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.990270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.990293 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.992822 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993352 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.993354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.993356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.993434 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993761 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.993765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.993766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.993963 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.995170 (+ 1207us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.995244 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.995368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260620 21:59:57.996369 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:57.984057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.17.67.252:46203)
0620 21:59:57.984152 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:57.984156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:57.984199 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:59:57.984421 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:57.984424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:57.984465 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:57.984582 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:57.984994 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.985704 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.986793 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:57.987122 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:57.987583 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:57.989762 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:57.989766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:57.989769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:57.989795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:57.992680 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993270 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.993274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.993275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.993363 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:57.993945 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:57.993950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:57.993951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:57.994171 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:57.995942 (+ 1771us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:57.996041 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 21:59:57.996172 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 21:59:57.998737 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: Bootstrap starting.
I20260620 21:59:58.000357 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: Bootstrap starting.
I20260620 21:59:58.001422 7934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:58.002300 7934 log.cc:824] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:58.002876 8009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: Neither blocks nor log segments found. Creating new log.
I20260620 21:59:58.004122 8009 log.cc:824] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:59:58.005049 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: No bootstrap required, opened a new log
I20260620 21:59:58.005517 8085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.005802 8085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f16a15a48f043b38bdc25e84f6b71d8, State: Initialized, Role: FOLLOWER
I20260620 21:59:58.006132 8085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.006507 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: No bootstrap required, opened a new log
I20260620 21:59:58.006731 8095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.006996 8095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.007509 8085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:58.008955 7934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.009172 7934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cce4b8d970499c86c58a659c85c2fe, State: Initialized, Role: FOLLOWER
I20260620 21:59:58.009501 7934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
W20260620 21:59:58.010826 7887 tablet.cc:2369] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:59:58.011122 7934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:58.011018 8009 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.011220 8009 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7c5e5c9d0142769d1b7e5c372dbfc9, State: Initialized, Role: FOLLOWER
I20260620 21:59:58.011312 7908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:59:58.011409 7908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:58.011546 8009 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.012372 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.012490 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.012717 7908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.012925 8009 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:59:58.014147 7982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02cce4b8d970499c86c58a659c85c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9"
I20260620 21:59:58.014355 7982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:58.015627 7982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02cce4b8d970499c86c58a659c85c2fe in term 1.
I20260620 21:59:58.016386 7884 leader_election.cc:302] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02cce4b8d970499c86c58a659c85c2fe, fc7c5e5c9d0142769d1b7e5c372dbfc9; no voters:
W20260620 21:59:58.016563 8119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:58.016626 8119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:58.017716 7908 leader_election.cc:288] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [CANDIDATE]: Term 1 election: Requested vote from peers fc7c5e5c9d0142769d1b7e5c372dbfc9 (127.17.67.253:42191), 5f16a15a48f043b38bdc25e84f6b71d8 (127.17.67.252:46203)
I20260620 21:59:58.020331 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02cce4b8d970499c86c58a659c85c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f16a15a48f043b38bdc25e84f6b71d8"
I20260620 21:59:58.020615 8059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:59:58.017616 7866 master_runner.cc:186] Error getting master registration for 127.17.67.253:42191: OK, instance_id { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" instance_seqno: 1781992797727872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:59:58.021315 7866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:59:58.022025 8059 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02cce4b8d970499c86c58a659c85c2fe in term 1.
I20260620 21:59:58.020028 8016 master_runner.cc:186] Error getting master registration for 127.17.67.254:39935: OK, instance_id { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" instance_seqno: 1781992797514258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:59:58.024011 8016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:59:58.024673 7941 master_runner.cc:263] No leader master found from master fc7c5e5c9d0142769d1b7e5c372dbfc9
I20260620 21:59:58.024750 7941 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 21:59:58.025058 8106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:58.025137 8106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 21:59:58.025106 8132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:59:58.025319 8132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:59:58.025377 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.025517 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.025629 8131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:59:58.025812 8131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [term 1 LEADER]: Becoming Leader. State: Replica: 02cce4b8d970499c86c58a659c85c2fe, State: Running, Role: LEADER
I20260620 21:59:58.026180 8131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } }
I20260620 21:59:58.028923 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cce4b8d970499c86c58a659c85c2fe. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.029068 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:58.030526 8143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:59:58.031464 8143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:59:58.037170 8059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [term 1 FOLLOWER]: Refusing update from remote peer 02cce4b8d970499c86c58a659c85c2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:58.037961 8107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:58.039916 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:58.025231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.17.67.254:39935)
0620 21:59:58.025391 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.025395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.026619 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 21:59:58.026875 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.026881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.026940 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.027026 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.027862 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.028606 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.029997 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.030395 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.030493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:58.035455 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:58.035459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:58.035462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:58.035490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:58.037248 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.037836 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.037839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.037841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.037928 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.038715 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.038718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.038720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.038937 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:58.039079 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:58.039651 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 21:59:58.039772 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:59:58.042882 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:58.025816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.17.67.253:42191)
0620 21:59:58.025952 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.025957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.031138 (+ 5181us) server_negotiation.cc:407] Connection header received
0620 21:59:58.031348 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.031355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.031423 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.031505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.032531 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.033137 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.034328 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.034667 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.034746 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:58.037435 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:58.037439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:58.037471 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:58.037502 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:58.039616 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.040527 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.040533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.040537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.040635 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.041008 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.041014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.041017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.041740 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:58.041860 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:58.042578 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 21:59:58.042715 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:59:58.043784 8095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cce4b8d970499c86c58a659c85c2fe. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.043921 8095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.045259 7982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [term 1 FOLLOWER]: Refusing update from remote peer 02cce4b8d970499c86c58a659c85c2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:59:58.045447 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:58.026586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.17.67.252:46203)
0620 21:59:58.026715 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.026721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.027195 (+ 474us) server_negotiation.cc:407] Connection header received
0620 21:59:58.027324 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.027330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.027385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.027457 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.028294 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.029105 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.030271 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:58.030639 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:58.030717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:58.040971 (+ 10254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:59:58.040975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:59:58.040978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:59:58.041007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:59:58.042921 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.043505 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.043509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.043511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.043630 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:59:58.044431 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:59:58.044435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:59:58.044439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:59:58.044682 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:59:58.044798 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:58.045150 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 21:59:58.045283 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:59:58.046818 8131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:59:58.051950 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cce4b8d970499c86c58a659c85c2fe. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.052129 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.060489 8107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.060637 8107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:58.061154 8108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.061296 8108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.066543 8145 mvcc.cc:205] Tried to move back new op lower bound from 7299042500752547840 to 7299042500721651712. Current Snapshot: MvccSnapshot[applied={T|T < 7299042500752547840}]
I20260620 21:59:58.066766 8095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.066884 8095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:59:58.067468 8107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02cce4b8d970499c86c58a659c85c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02cce4b8d970499c86c58a659c85c2fe" member_type: VOTER last_known_addr { host: "127.17.67.254" port: 39935 } } peers { permanent_uuid: "fc7c5e5c9d0142769d1b7e5c372dbfc9" member_type: VOTER last_known_addr { host: "127.17.67.253" port: 42191 } } peers { permanent_uuid: "5f16a15a48f043b38bdc25e84f6b71d8" member_type: VOTER last_known_addr { host: "127.17.67.252" port: 46203 } } }
I20260620 21:59:58.067598 8107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe [sys.catalog]: This master's current role is: LEADER
I20260620 21:59:58.071956 8143 catalog_manager.cc:1347] Generated new cluster ID: 56a963df69f64bf083b5922d98f47e74
I20260620 21:59:58.072055 8143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:59:58.104722 8143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:59:58.106199 8143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:59:58.122974 8143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02cce4b8d970499c86c58a659c85c2fe: Generated new TSK 0
I20260620 21:59:58.124984 8143 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:59:58.157363 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.154199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.17.67.253:42191)
0620 21:59:58.154340 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.154345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.156183 (+ 1838us) server_negotiation.cc:407] Connection header received
0620 21:59:58.156458 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.156462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.156514 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.156569 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.157207 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52782: BlockingRecv error: recv got EOF from 127.0.0.1:52782 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 21:59:58.159011 8091 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.154945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.17.67.252:46203)
0620 21:59:58.155056 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.155062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.157758 (+ 2696us) server_negotiation.cc:407] Connection header received
0620 21:59:58.158265 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.158268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.158322 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.158373 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.158893 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50578: BlockingRecv error: recv got EOF from 127.0.0.1:50578 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 21:59:58.160075 8090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.155554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.17.67.254:39935)
0620 21:59:58.155673 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.155679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.158170 (+ 2491us) server_negotiation.cc:407] Connection header received
0620 21:59:58.159372 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.159376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.159427 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.159477 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.159951 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45908: BlockingRecv error: recv got EOF from 127.0.0.1:45908 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 21:59:58.164535 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.163069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.17.67.253:42191)
0620 21:59:58.163168 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.163172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.163586 (+ 414us) server_negotiation.cc:407] Connection header received
0620 21:59:58.163797 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.163801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.163874 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.163925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.164438 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52798: BlockingRecv error: recv got EOF from 127.0.0.1:52798 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 21:59:58.169534 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.167872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.17.67.253:42191)
0620 21:59:58.167961 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.167965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.168557 (+ 592us) server_negotiation.cc:407] Connection header received
0620 21:59:58.168760 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.168763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.168807 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.168854 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.169442 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52800: BlockingRecv error: recv got EOF from 127.0.0.1:52800 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 21:59:58.174695 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.173320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.17.67.253:42191)
0620 21:59:58.173411 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.173415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.173895 (+ 480us) server_negotiation.cc:407] Connection header received
0620 21:59:58.174067 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.174070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.174114 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.174161 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.174609 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52810: BlockingRecv error: recv got EOF from 127.0.0.1:52810 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 21:59:58.182097 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.180623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.17.67.253:42191)
0620 21:59:58.180747 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.180751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.181243 (+ 492us) server_negotiation.cc:407] Connection header received
0620 21:59:58.181426 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.181429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.181474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.181521 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.182005 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52824: BlockingRecv error: recv got EOF from 127.0.0.1:52824 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 21:59:58.193775 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.192136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.17.67.253:42191)
0620 21:59:58.192253 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.192257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.192797 (+ 540us) server_negotiation.cc:407] Connection header received
0620 21:59:58.192995 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.192998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.193045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.193094 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.193672 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52836: BlockingRecv error: recv got EOF from 127.0.0.1:52836 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 21:59:58.214322 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.212335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.17.67.253:42191)
0620 21:59:58.212470 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.212474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.213109 (+ 635us) server_negotiation.cc:407] Connection header received
0620 21:59:58.213420 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.213425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.213487 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.213551 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.214171 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52842: BlockingRecv error: recv got EOF from 127.0.0.1:52842 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 21:59:58.250312 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.248864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.17.67.253:42191)
0620 21:59:58.248975 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.248979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.249360 (+ 381us) server_negotiation.cc:407] Connection header received
0620 21:59:58.249527 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.249531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.249627 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.249719 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.250181 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52854: BlockingRecv error: recv got EOF from 127.0.0.1:52854 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 21:59:58.317772 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.316587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.17.67.253:42191)
0620 21:59:58.316755 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.316759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.316840 (+ 81us) server_negotiation.cc:407] Connection header received
0620 21:59:58.316980 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.316983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.317034 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.317115 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.317654 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52866: BlockingRecv error: recv got EOF from 127.0.0.1:52866 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 21:59:58.449541 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.448005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.17.67.253:42191)
0620 21:59:58.448239 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.448245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.448476 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:59:58.448708 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.448712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.448766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.448874 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.449423 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52870: BlockingRecv error: recv got EOF from 127.0.0.1:52870 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 21:59:58.709646 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:59:58.708247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.17.67.253:42191)
0620 21:59:58.708398 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:58.708403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:58.708522 (+ 119us) server_negotiation.cc:407] Connection header received
0620 21:59:58.708717 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:58.708721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:58.708784 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:58.708905 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:59:58.709493 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52880: BlockingRecv error: recv got EOF from 127.0.0.1:52880 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:59:59.026590 8106 catalog_manager.cc:1259] Loaded cluster ID: 56a963df69f64bf083b5922d98f47e74
I20260620 21:59:59.026656 8132 catalog_manager.cc:1259] Loaded cluster ID: 56a963df69f64bf083b5922d98f47e74
I20260620 21:59:59.026697 8106 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: loading cluster ID for follower catalog manager: success
I20260620 21:59:59.026714 8132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: loading cluster ID for follower catalog manager: success
I20260620 21:59:59.028388 8106 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: acquiring CA information for follower catalog manager: success
I20260620 21:59:59.028422 8132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: acquiring CA information for follower catalog manager: success
I20260620 21:59:59.029520 8132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc7c5e5c9d0142769d1b7e5c372dbfc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:59:59.029520 8106 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f16a15a48f043b38bdc25e84f6b71d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:59:59.228112 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:59.223863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.17.67.253:42191)
0620 21:59:59.224146 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:59.224152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:59.224220 (+ 68us) server_negotiation.cc:407] Connection header received
0620 21:59:59.224410 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:59.224414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:59.224546 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:59.224668 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:59.224955 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.225531 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.226590 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.226825 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.226938 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:59.227260 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:59.227691 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:59.227834 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:59.227896 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:59:59.227952 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":65,"threads_started":1}
I20260620 21:59:59.235881 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:59.231257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.17.67.252:46203)
0620 21:59:59.231521 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:59.231529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:59.231545 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:59:59.231757 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:59.231760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:59.231890 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:59.232030 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:59.232268 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.232847 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.233852 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.234333 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.234447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:59.234926 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:59.235455 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:59.235560 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:59.235692 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 21:59:59.235744 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260620 21:59:59.243695 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:59.239384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.17.67.253:42191)
0620 21:59:59.239530 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:59.239534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:59.239943 (+ 409us) server_negotiation.cc:407] Connection header received
0620 21:59:59.240148 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:59.240151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:59.240209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:59.240281 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:59.240568 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.241158 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.242367 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.242749 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.242845 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:59.243169 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:59.243263 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:59.243310 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:59.243475 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 21:59:59.243546 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:59:59.244735 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:59.239893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.17.67.252:46203)
0620 21:59:59.240006 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:59.240010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:59.240243 (+ 233us) server_negotiation.cc:407] Connection header received
0620 21:59:59.240414 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:59.240420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:59.240479 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:59.240544 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:59.240896 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.241698 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.243374 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.243794 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.243874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:59.244229 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:59.244307 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:59.244383 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:59.244516 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 21:59:59.244578 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:59:59.245038 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:59:59.239420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.17.67.254:39935)
0620 21:59:59.239699 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 21:59:59.239703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:59:59.240021 (+ 318us) server_negotiation.cc:407] Connection header received
0620 21:59:59.240139 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:59:59.240142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:59:59.240354 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:59:59.240475 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:59:59.240707 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.241542 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.242773 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:59:59.243099 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:59:59.243203 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:59:59.243627 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:59:59.244499 (+ 872us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:59:59.244579 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:59:59.244869 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:59:59.244933 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 21:59:59.248790 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7866
I20260620 21:59:59.255267 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 7941
I20260620 21:59:59.261125 17679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiPMPYE/build/debug/bin/kudu with pid 8016
2026-06-20T21:59:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3079 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6192 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286010 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST